[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 728970697 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2512MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006340] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.009164] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011044] SELinux: Initializing. [ 0.012086] *** VALIDATE selinux *** [ 0.020316] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025298] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] *** VALIDATE tmpfs *** [ 0.029468] *** VALIDATE proc *** [ 0.030320] *** VALIDATE cgroup *** [ 0.031011] *** VALIDATE cgroup2 *** [ 0.032298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035034] Spectre V2 : User space: Vulnerable [ 0.036019] Speculative Store Bypass: Vulnerable [ 0.039422] debug: unmapping init [mem 0xffffffffb8459000-0xffffffffb8460fff] [ 0.041301] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042869] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043033] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050393] rcu: Hierarchical SRCU implementation. [ 0.053149] smp: Bringing up secondary CPUs ... [ 0.054788] x86: Booting SMP configuration: [ 0.055048] .... node #0, CPUs: #1 #2 #3 [ 0.116150] smp: Brought up 1 node, 4 CPUs [ 0.118013] smpboot: Max logical packages: 1 [ 0.119012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235395] node 0 deferred pages initialised in 47ms [ 0.239156] devtmpfs: initialized [ 0.240203] x86/mm: Memory block size: 128MB [ 0.242817] gcov: version magic: 0x41383552 [ 0.244082] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246213] pinctrl core: initialized pinctrl subsystem [ 0.247157] [ 0.247615] ************************************************************* [ 0.248011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249010] ** ** [ 0.250008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251007] ** ** [ 0.252008] ** This means that this kernel is built to expose internal ** [ 0.253011] ** IOMMU data structures, which may compromise security on ** [ 0.254008] ** your system. ** [ 0.255010] ** ** [ 0.256007] ** If you see this message and you are not debugging the ** [ 0.257010] ** kernel, report this immediately to your vendor! ** [ 0.258012] ** ** [ 0.259012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260011] ************************************************************* [ 0.261894] NET: Registered protocol family 16 [ 0.262655] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.266633] cpuidle: using governor menu [ 0.270816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.271512] PCI: Using configuration type 1 for base access [ 0.272130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.284164] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287526] cryptd: max_cpu_qlen set to 1000 [ 0.291604] ACPI: Added _OSI(Module Device) [ 0.292028] ACPI: Added _OSI(Processor Device) [ 0.293012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.294009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299698] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.303423] ACPI: Interpreter enabled [ 0.304125] ACPI: PM: (supports S0 S3 S4 S5) [ 0.305011] ACPI: Using IOAPIC for interrupt routing [ 0.306449] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.307292] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.315398] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318158] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.320962] acpiphp: Slot [2] registered [ 0.321109] acpiphp: Slot [5] registered [ 0.322104] acpiphp: Slot [6] registered [ 0.323076] acpiphp: Slot [7] registered [ 0.324133] acpiphp: Slot [8] registered [ 0.325092] acpiphp: Slot [9] registered [ 0.326119] acpiphp: Slot [10] registered [ 0.327067] acpiphp: Slot [3] registered [ 0.328012] acpiphp: Slot [4] registered [ 0.329131] acpiphp: Slot [11] registered [ 0.330076] acpiphp: Slot [12] registered [ 0.331095] acpiphp: Slot [13] registered [ 0.332106] acpiphp: Slot [14] registered [ 0.333079] acpiphp: Slot [15] registered [ 0.334117] acpiphp: Slot [16] registered [ 0.335091] acpiphp: Slot [17] registered [ 0.336110] acpiphp: Slot [18] registered [ 0.337072] acpiphp: Slot [19] registered [ 0.338079] acpiphp: Slot [20] registered [ 0.339129] acpiphp: Slot [21] registered [ 0.340095] acpiphp: Slot [22] registered [ 0.341081] acpiphp: Slot [23] registered [ 0.342082] acpiphp: Slot [24] registered [ 0.343075] acpiphp: Slot [25] registered [ 0.344066] acpiphp: Slot [26] registered [ 0.345068] acpiphp: Slot [27] registered [ 0.346083] acpiphp: Slot [28] registered [ 0.347100] acpiphp: Slot [29] registered [ 0.348138] acpiphp: Slot [30] registered [ 0.349077] acpiphp: Slot [31] registered [ 0.350038] PCI host bridge to bus 0000:00 [ 0.351012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.352021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.353016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.354042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359544] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362624] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369758] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372621] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376589] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381590] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397072] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402277] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422495] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440899] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.443013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.446017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.453062] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.458905] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.461013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.464012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.471024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.476956] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.482018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.489016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.495094] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.498014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.501014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.508016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.515407] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.516367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.517325] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.518398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.519211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.527042] iommu: Default domain type: Passthrough [ 0.528443] SCSI subsystem initialized [ 0.529119] ACPI: bus type USB registered [ 0.530097] usbcore: registered new interface driver usbfs [ 0.531050] usbcore: registered new interface driver hub [ 0.532064] usbcore: registered new device driver usb [ 0.533169] pps_core: LinuxPPS API ver. 1 registered [ 0.534008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.535052] PTP clock support registered [ 0.536183] EDAC MC: Ver: 3.0.0 [ 0.537471] PCI: Using ACPI for IRQ routing [ 0.538868] NetLabel: Initializing [ 0.539008] NetLabel: domain hash size = 128 [ 0.540008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541148] NetLabel: unlabeled traffic allowed by default [ 0.550067] vgaarb: loaded [ 0.556384] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.562015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.575000] clocksource: Switched to clocksource kvm-clock [ 0.688120] VFS: Disk quotas dquot_6.6.0 [ 0.689291] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.691964] *** VALIDATE ramfs *** [ 0.693407] *** VALIDATE hugetlbfs *** [ 0.695398] pnp: PnP ACPI init [ 0.698141] pnp: PnP ACPI: found 6 devices [ 0.715199] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.718947] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721466] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.723865] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.726723] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.729598] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.732957] NET: Registered protocol family 2 [ 0.735702] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.742228] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.746087] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.752095] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.759403] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.762966] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766466] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.769863] NET: Registered protocol family 1 [ 0.774137] RPC: Registered named UNIX socket transport module. [ 0.776476] RPC: Registered udp transport module. [ 0.778855] RPC: Registered tcp transport module. [ 0.780945] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.783902] NET: Registered protocol family 44 [ 0.785554] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.803702] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 17719 usecs [ 0.806425] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.808806] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.811520] PCI: CLS 0 bytes, default 64 [ 0.813577] Unpacking initramfs... [ 2.666436] debug: unmapping init [mem 0xffff936a7cc54000-0xffff936a7ffbffff] [ 2.670884] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.673617] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.677323] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.819182] Initialise system trusted keyrings [ 3.822062] Key type blacklist registered [ 3.823822] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.844668] zbud: loaded [ 3.848023] *** VALIDATE nfs *** [ 3.849178] *** VALIDATE nfs4 *** [ 3.851378] pstore: using deflate compression [ 3.856553] Platform Keyring initialized [ 4.030409] NET: Registered protocol family 38 [ 4.033576] Key type asymmetric registered [ 4.036186] Asymmetric key parser 'x509' registered [ 4.042672] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.046705] io scheduler mq-deadline registered [ 4.048689] io scheduler kyber registered [ 4.050315] io scheduler bfq registered [ 4.052334] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.061260] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.068455] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.072519] ACPI: Power Button [PWRF] [ 4.079632] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.090717] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.118791] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.136873] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.175178] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.207248] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.236966] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.242368] Non-volatile memory driver v1.3 [ 4.243626] Linux agpgart interface v0.103 [ 4.281597] virtio_blk virtio1: [vda] 145176 512-byte logical blocks (74.3 MB/70.9 MiB) [ 4.284056] vda: detected capacity change from 0 to 74330112 [ 4.301355] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.304658] vdb: detected capacity change from 0 to 1073741824 [ 4.324232] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.326981] vdc: detected capacity change from 0 to 2621440000 [ 4.345223] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.348224] vdd: detected capacity change from 0 to 2621440000 [ 4.374260] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.376862] vde: detected capacity change from 0 to 4294967296 [ 4.399295] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.416773] vdf: detected capacity change from 0 to 4294967296 [ 4.437813] libphy: Fixed MDIO Bus: probed [ 4.445293] usbcore: registered new interface driver usbserial_generic [ 4.448544] usbserial: USB Serial support registered for generic [ 4.450647] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.455688] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.457282] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.459544] mousedev: PS/2 mouse device common for all mice [ 4.462728] rtc_cmos 00:05: RTC can wake from S4 [ 4.465231] rtc_cmos 00:05: registered as rtc0 [ 4.467524] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.471817] intel_pstate: CPU model not supported [ 4.474739] hid: raw HID events driver (C) Jiri Kosina [ 4.476423] usbcore: registered new interface driver usbhid [ 4.479456] usbhid: USB HID core driver [ 4.481221] drop_monitor: Initializing network drop monitor service [ 4.483872] Initializing XFRM netlink socket [ 4.485735] NET: Registered protocol family 10 [ 4.490352] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.492635] Segment Routing with IPv6 [ 4.564135] NET: Registered protocol family 17 [ 4.566071] mpls_gso: MPLS GSO support [ 4.646806] RAS: Correctable Errors collector initialized. [ 4.650043] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.650867] AVX version of gcm_enc/dec engaged. [ 4.667408] AES CTR mode by8 optimization enabled [ 4.670880] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.761370] sched_clock: Marking stable (4761294787, 0)->(6968895290, -2207600503) [ 4.765094] registered taskstats version 1 [ 4.766974] Loading compiled-in X.509 certificates [ 4.769435] zswap: loaded using pool lzo/zbud [ 4.793882] Key type big_key registered [ 4.809969] Key type encrypted registered [ 4.811176] ima: No TPM chip found, activating TPM-bypass! [ 4.812618] ima: Allocated hash algorithm: sha1 [ 4.814259] ima: No architecture policies found [ 4.815608] evm: Initialising EVM extended attributes: [ 4.817228] evm: security.selinux [ 4.818182] evm: security.ima [ 4.818889] evm: security.capability [ 4.819774] evm: HMAC attrs: 0x1 [ 4.822207] rtc_cmos 00:05: setting system clock to 2026-07-01 17:40:58 UTC (1782927658) [ 4.830893] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 4.833587] debug: unmapping init [mem 0xffffffffb8182000-0xffffffffb8458fff] [ 4.842156] Write protecting the kernel read-only data: 28672k [ 4.845576] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 4.847842] debug: unmapping init [mem 0xffffffffb7114000-0xffffffffb71fffff] [ 4.885506] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.902738] systemd[1]: Detected virtualization kvm. [ 4.907359] systemd[1]: Detected architecture x86-64. [ 4.910192] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.946165] systemd[1]: No hostname configured. [ 4.948314] systemd[1]: Set hostname to . [ 4.952273] random: systemd: uninitialized urandom read (16 bytes read) [ 4.957020] systemd[1]: Initializing machine ID from random generator. [ 5.119798] random: ln: uninitialized urandom read (6 bytes read) [ 5.508843] random: systemd: uninitialized urandom read (16 bytes read) [ 5.517787] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.583067] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.604888] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.106860] device-mapper: uevent: version 1.0.3 [ 7.112455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.168100] virtio_net virtio0 ens2: renamed from eth0 [ 9.737545] random: fast init done [ 10.165020] scsi host0: ata_piix [ 10.268750] scsi host1: ata_piix [ 10.277785] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.294253] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.335622] random: crng init done [ 15.337215] random: 7 urandom warning(s) missed due to ratelimiting [ 17.696971] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.504485] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.975547] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.696093] SELinux: Disabled at runtime. [ 22.778462] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.791866] systemd[1]: Detected virtualization kvm. [ 22.793810] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.035338] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.042319] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.050890] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.060322] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.067323] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.075869] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.088510] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ 24.139852] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.848636] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.736229] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.383698] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.161479] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.280795] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.693245] Key type dns_resolver registered [ ***] A start job is running for Configur…-only root support (9s / no limit)[ 34.242482] NFS: Registering the id_resolver key type [ 34.245954] Key type id_resolver registered [ 34.252316] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 61.289048] libcfs: loading out-of-tree module taints kernel. [ 61.309579] Key type ._llcrypt registered [ 61.311158] Key type .llcrypt registered [ 61.355958] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 68.614245] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 69.146902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.152077] alg: No test for adler32 (adler32-zlib) [ 70.129139] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 70.405914] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 72.007158] Key type lgssc registered [ 72.577413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.988752] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.009319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 98.374817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 98.382986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.478171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 99.489951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 99.530776] Lustre: lustre-MDT0000: new disk, initializing [ 99.559186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.567086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 100.960242] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.319987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 106.361261] Lustre: 6490:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 106.379092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 106.381506] Lustre: Skipped 1 previous similar message [ 106.425706] Lustre: lustre-MDT0001: new disk, initializing [ 106.451989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 106.462213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 106.464815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 107.865801] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.137500] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.381932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 113.478833] Lustre: lustre-OST0000: new disk, initializing [ 113.481262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.484463] Lustre: 8428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 113.503214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.115977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 115.118539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 115.143518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.482154] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.245210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 121.298488] Lustre: lustre-OST0001: new disk, initializing [ 121.301601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.305793] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 121.328596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.349506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 121.354313] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 121.392163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.472663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.366317] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 134.080158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.477431] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 138.252038] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 140.021367] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 140.969044] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 141.928250] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 142.561163] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 1 13:43:15 EDT 2026 [ 148.290389] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 148.788818] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 162.271848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.272179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.275605] Lustre: Skipped 3 previous similar messages [ 162.277679] Lustre: Skipped 2 previous similar messages [ 166.619644] Lustre: server umount lustre-MDT0000 complete [ 167.840707] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782927821 with bad export cookie 7079209960693075127 [ 167.841828] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.845836] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 172.512797] LustreError: 6496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.513073] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.513386] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.513391] Lustre: Skipped 1 previous similar message [ 172.517360] LustreError: 6496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 172.533063] Lustre: Skipped 1 previous similar message [ 177.631534] LustreError: 10750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.631630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 177.635691] LustreError: 10750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 177.637247] Lustre: Skipped 2 previous similar messages [ 178.655639] LustreError: 6496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 178.661014] LustreError: 6496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 182.239194] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 182.305120] Lustre: server umount lustre-MDT0001 complete [ 189.696240] Lustre: server umount lustre-OST0000 complete [ 205.279106] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 205.327900] Lustre: server umount lustre-OST0001 complete [ 207.544287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 210.859470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 223.113715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.192649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 223.202598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.235337] Lustre: lustre-MDT0000: new disk, initializing [ 223.255178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.261865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.528638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 228.612206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.656588] Lustre: 15040:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.761955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.075577] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 233.708779] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.398184] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 237.527332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.618257] Lustre: lustre-OST0000: new disk, initializing [ 237.619344] Lustre: Skipped 1 previous similar message [ 237.620728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.622135] Lustre: Skipped 2 previous similar messages [ 237.623803] Lustre: 17008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.653717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.655682] Lustre: Skipped 1 previous similar message [ 239.288827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.292305] Lustre: Skipped 1 previous similar message [ 239.294617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.329785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.451199] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 243.046451] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 244.388933] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 244.462871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 245.731259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 245.798327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 249.824644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 249.825194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.827495] LustreError: Skipped 1 previous similar message [ 249.831722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 249.835164] Lustre: Skipped 3 previous similar messages [ 252.491595] Lustre: server umount lustre-OST0000 complete [ 254.431808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.433968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.437272] Lustre: Skipped 1 previous similar message [ 254.438736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.184205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.186697] Lustre: Skipped 4 previous similar messages [ 268.255148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 268.323209] Lustre: server umount lustre-MDT0000 complete [ 269.564240] LustreError: 17007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782927923 with bad export cookie 7079209960693076646 [ 269.567731] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.567907] LustreError: 17007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 269.674800] Lustre: server umount lustre-MDT0001 complete [ 271.031129] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:45:24 (1782927924) [ 271.857304] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 272.406985] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:45:25 (1782927925) [ 274.591888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.757110] LustreError: 19417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.780437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.179204] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 279.166776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.575134] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 282.938987] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 284.762198] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 287.012215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.008740] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 291.294195] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 304.609031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.610417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.611254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.612179] LustreError: Skipped 1 previous similar message [ 304.621432] Lustre: Skipped 2 previous similar messages [ 310.545897] Lustre: server umount lustre-OST0000 complete [ 314.848751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.043242] Lustre: server umount lustre-MDT0000 complete [ 319.360449] LustreError: 19399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782927973 with bad export cookie 7079209960693077528 [ 319.365137] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.365153] LustreError: 19399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.484983] Lustre: server umount lustre-MDT0001 complete [ 323.204884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 324.258358] Key type lgssc unregistered [ 324.398340] LNet: 22968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.400828] LNetError: 22968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.412266] LNet: Removed LNI 192.168.204.144@tcp [ 324.772126] Key type .llcrypt unregistered [ 324.773662] Key type ._llcrypt unregistered [ 333.609463] Key type ._llcrypt registered [ 333.610802] Key type .llcrypt registered [ 333.651091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 333.979607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 334.027272] alg: No test for adler32 (adler32-zlib) [ 334.882289] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 334.968365] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 336.543150] Key type lgssc registered [ 336.915492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 339.616789] loop: module loaded [ 339.637493] loop0: detected capacity change from 0 to 204800000 [ 339.714642] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.718768] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.831282] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.802271] loop0: detected capacity change from 0 to 204800000 [ 341.854335] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.859450] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.864721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.523989] loop0: detected capacity change from 0 to 204800000 [ 345.281034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 345.286151] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.366125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.378345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.411950] Lustre: lustre-MDT0000: new disk, initializing [ 346.432811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.439177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 347.745545] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 351.305734] loop1: detected capacity change from 0 to 204800000 [ 353.319454] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.425631] Lustre: lustre-OST07c6: new disk, initializing [ 353.427556] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 353.429841] Lustre: Skipped 1 previous similar message [ 353.432294] Lustre: 26133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 353.458444] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 355.406268] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 356.289682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 356.295678] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 356.307459] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 357.537520] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 357.541123] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 357.544763] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 357.547253] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 357.550590] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 357.554464] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 373.727244] Lustre: 25119:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 373.731782] Lustre: 25119:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 373.734461] Lustre: 25119:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 373.737970] Lustre: 25119:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 373.741666] Lustre: 25119:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 373.744431] Lustre: 25119:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 377.218573] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 377.221267] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 377.223427] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 377.225700] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.228319] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 377.231107] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.233226] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 377.236746] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.240206] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 377.243310] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.246504] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 377.248432] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.303250] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 380.307019] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 380.310025] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 380.312591] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.315380] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 380.318018] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.320309] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 380.323189] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.326169] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 380.329103] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.332054] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 380.334497] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 386.881120] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 386.883575] Lustre: 26141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 386.885583] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 386.887502] Lustre: 26141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 386.889620] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 386.891603] Lustre: 26141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 386.893564] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 386.895608] Lustre: 26141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 386.897468] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 386.899197] Lustre: 26141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 386.901057] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 386.902738] Lustre: 26141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 396.852457] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 396.855025] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 396.857772] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 396.860638] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 396.863630] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 396.866371] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 396.869013] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 396.871955] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 396.874827] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 396.876905] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 396.879044] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 396.881489] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 438.240539] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.244168] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.250775] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 443.360493] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 443.979118] Lustre: server umount lustre-OST07c6 complete [ 447.183487] Lustre: server umount lustre-MDT0000 complete [ 452.731838] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 453.739406] Key type lgssc unregistered [ 453.860372] LNet: 28704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.863766] LNetError: 28704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 453.876322] LNet: Removed LNI 192.168.204.144@tcp [ 454.169045] Key type .llcrypt unregistered [ 454.170278] Key type ._llcrypt unregistered [ 461.676076] Key type ._llcrypt registered [ 461.677863] Key type .llcrypt registered [ 461.715206] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 462.154715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 462.165452] alg: No test for adler32 (adler32-zlib) [ 463.027291] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 463.122812] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 464.703193] Key type lgssc registered [ 465.133602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 467.773473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 467.781321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.929063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.300251] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 473.176797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.310532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.689443] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 477.137538] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.023183] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.344264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 481.460693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.422243] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 485.895034] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.730386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 503.263761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.265902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.269685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.777388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 503.779513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.783326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.086776] Lustre: server umount lustre-OST0000 complete [ 509.408709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.409233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.410400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.410405] Lustre: Skipped 2 previous similar messages [ 509.422808] Lustre: Skipped 1 previous similar message [ 514.528797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.531710] Lustre: Skipped 2 previous similar messages [ 514.592109] Lustre: server umount lustre-MDT0000 complete [ 515.900246] LustreError: 31792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928169 with bad export cookie 4046002229012485622 [ 515.903466] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.904248] LustreError: 31792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 516.015046] Lustre: server umount lustre-MDT0001 complete [ 519.854425] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 520.991698] Key type lgssc unregistered [ 521.127678] LNet: 33704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.130855] LNetError: 33704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.139566] LNet: Removed LNI 192.168.204.144@tcp [ 521.491868] Key type .llcrypt unregistered [ 521.493128] Key type ._llcrypt unregistered [ 529.100872] Key type ._llcrypt registered [ 529.102556] Key type .llcrypt registered [ 529.139469] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 529.550334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 529.588797] alg: No test for adler32 (adler32-zlib) [ 530.441383] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 530.532185] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 532.111135] Key type lgssc registered [ 532.501348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 535.255864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 535.263460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.416673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.892972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 540.966046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.095290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 542.504742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 545.041322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 546.937751] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.359263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.487546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.534268] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 554.130911] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 561.763609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 571.359753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.362240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.368049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 572.386064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.388433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.388848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 575.952868] Lustre: server umount lustre-OST0000 complete [ 577.504381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.505129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.508620] Lustre: Skipped 1 previous similar message [ 577.511339] Lustre: Skipped 1 previous similar message [ 582.624037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 582.626484] Lustre: Skipped 1 previous similar message [ 583.379453] Lustre: server umount lustre-MDT0000 complete [ 584.691486] LustreError: 34931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928238 with bad export cookie 5202963623441901998 [ 584.695248] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.696654] LustreError: 34931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 584.804850] Lustre: server umount lustre-MDT0001 complete [ 588.489867] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 589.553476] Key type lgssc unregistered [ 589.676349] LNet: 38498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.679769] LNetError: 38498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.688412] LNet: Removed LNI 192.168.204.144@tcp [ 590.004773] Key type .llcrypt unregistered [ 590.006431] Key type ._llcrypt unregistered [ 599.312209] Key type ._llcrypt registered [ 599.313358] Key type .llcrypt registered [ 599.363319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 599.767986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 599.784046] alg: No test for adler32 (adler32-zlib) [ 600.640110] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 600.735345] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 602.319157] Key type lgssc registered [ 602.707855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 605.353544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 605.359245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.505940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.859024] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 610.844666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.963577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 612.392347] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 614.711518] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 616.492519] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 618.776532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.886435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.752381] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 623.044718] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 633.255477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 636.895716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.898237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.901698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.432328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.433567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.816985] Lustre: server umount lustre-MDT0000 complete [ 642.994292] LustreError: 40626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928296 with bad export cookie 12914861119953687950 [ 642.997186] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.999498] LustreError: 40626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 643.551761] LustreError: 39890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.552072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.555571] LustreError: 39890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 643.558682] Lustre: Skipped 1 previous similar message [ 643.562400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 643.564059] Lustre: Skipped 3 previous similar messages [ 647.583831] LustreError: 42379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.584073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 647.589961] LustreError: 42379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 647.592390] Lustre: Skipped 1 previous similar message [ 648.860952] LustreError: 39888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.704384] LustreError: 42379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.704441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 652.710855] Lustre: Skipped 1 previous similar message [ 657.375141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 657.453191] Lustre: server umount lustre-MDT0001 complete [ 675.274390] Lustre: server umount lustre-OST0000 complete [ 680.551812] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 681.549931] Key type lgssc unregistered [ 681.670338] LNet: 43773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.673641] LNetError: 43773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.687308] LNet: Removed LNI 192.168.204.144@tcp [ 681.986870] Key type .llcrypt unregistered [ 681.988458] Key type ._llcrypt unregistered [ 690.503196] Key type ._llcrypt registered [ 690.504372] Key type .llcrypt registered [ 690.551270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 690.981753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 691.007424] alg: No test for adler32 (adler32-zlib) [ 691.853099] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 691.936359] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 693.511121] Key type lgssc registered [ 693.873592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.343811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.349669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.483806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.679674] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 701.292220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.414256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 702.726793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 704.976201] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 706.717242] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 708.874990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.985968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.878788] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 713.221658] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 723.302714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 727.519992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 727.522194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.526354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.546658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.549318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.549510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.664799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 733.666543] Lustre: Skipped 3 previous similar messages [ 738.785136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.786933] Lustre: Skipped 2 previous similar messages [ 740.319099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 740.368875] Lustre: server umount lustre-OST0000 complete [ 752.978955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.984235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.985648] Lustre: Skipped 1 previous similar message [ 757.727735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 757.730311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.187920] Lustre: server umount lustre-MDT0000 complete [ 760.317080] LustreError: 45147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928413 with bad export cookie 11426594944979969285 [ 760.319857] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.321389] LustreError: 45147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 766.501234] Lustre: server umount lustre-MDT0001 complete [ 769.783421] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 770.713707] Key type lgssc unregistered [ 770.845422] LNet: 48890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.847903] LNetError: 48890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.860602] LNet: Removed LNI 192.168.204.144@tcp [ 771.166306] Key type .llcrypt unregistered [ 771.167969] Key type ._llcrypt unregistered [ 779.494184] Key type ._llcrypt registered [ 779.495022] Key type .llcrypt registered [ 779.526078] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 779.893940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.900870] alg: No test for adler32 (adler32-zlib) [ 780.736319] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 780.815738] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 782.391093] Key type lgssc registered [ 782.729102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 785.237184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 785.244016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.378383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.643064] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 790.301169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.414458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 791.686211] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 793.918250] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 795.697930] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 797.911714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 798.017181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.852631] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 802.124230] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 811.365676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 814.638833] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 815.651353] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 826.288020] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-0113h54m39s [ 826.757175] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 828.067402] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 829.084655] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 831.455842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 831.459078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.464518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.555789] Lustre: server umount lustre-MDT0000 complete [ 834.581786] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 835.228577] LustreError: 50308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.236636] LustreError: 50308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 836.685853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.721897] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 836.803688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.828811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 837.860900] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 838.041883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 839.133049] Lustre: DEBUG MARKER: Wait for df (36387) ... [ 839.624822] Lustre: DEBUG MARKER: done [ 842.208292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 842.208399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.211589] LustreError: Skipped 1 previous similar message [ 842.215177] Lustre: Skipped 2 previous similar messages [ 842.217201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 842.220637] Lustre: Skipped 6 previous similar messages [ 846.601242] Lustre: server umount lustre-OST0000 complete [ 848.352741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 848.352885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.353429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 848.353434] Lustre: Skipped 2 previous similar messages [ 848.362057] Lustre: Skipped 1 previous similar message [ 860.640497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 860.642345] Lustre: Skipped 7 previous similar messages [ 862.175091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 862.228139] Lustre: server umount lustre-MDT0000 complete [ 863.339243] LustreError: 50264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928517 with bad export cookie 8043973558711764160 [ 863.340995] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.343634] LustreError: 50264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 863.449304] Lustre: server umount lustre-MDT0001 complete [ 866.688273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 867.596277] Key type lgssc unregistered [ 867.722250] LNet: 55215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.724969] LNetError: 55215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.731271] LNet: Removed LNI 192.168.204.144@tcp [ 868.027381] Key type .llcrypt unregistered [ 868.028474] Key type ._llcrypt unregistered [ 881.419512] Key type ._llcrypt registered [ 881.420446] Key type .llcrypt registered [ 881.455446] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 881.799847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.833963] alg: No test for adler32 (adler32-zlib) [ 882.678388] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 882.760838] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 884.335157] Key type lgssc registered [ 884.698852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.171426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.177979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.257719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 888.265533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 888.323709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.590015] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 892.401147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.433453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 892.443498] Lustre: 57400:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 892.529284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.829579] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 896.152951] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.876403] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 900.120575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.189183] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 900.250946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.193071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 902.025946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 904.218942] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 907.404318] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-0113h56m00s [ 907.858353] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 909.892676] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 910.637725] Lustre: DEBUG MARKER: Creating a test file and stat it [ 910.650523] LustreError: 58661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 930.985827] Lustre: lustre-MDT0000: Client 5a114c3f-d46c-48f1-9689-9a718653b0be (at 192.168.204.44@tcp) reconnecting [ 950.695085] LustreError: 58661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 951.411626] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 952.229434] Lustre: DEBUG MARKER: done [ 956.897073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 956.899333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.903131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.409231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.410818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.414624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.884992] Lustre: server umount lustre-OST0000 complete [ 962.528732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.529528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.532578] Lustre: Skipped 1 previous similar message [ 962.536093] Lustre: Skipped 2 previous similar messages [ 967.258627] Lustre: server umount lustre-MDT0000 complete [ 968.503143] LustreError: 60971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928622 with bad export cookie 3573297334677724316 [ 968.505438] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.506918] LustreError: 60971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 968.612974] Lustre: server umount lustre-MDT0001 complete [ 971.900587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 972.796912] Key type lgssc unregistered [ 972.913276] LNet: 61722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.915705] LNetError: 61722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.922253] LNet: Removed LNI 192.168.204.144@tcp [ 973.192872] Key type .llcrypt unregistered [ 973.193910] Key type ._llcrypt unregistered [ 987.616186] Key type ._llcrypt registered [ 987.617718] Key type .llcrypt registered [ 987.663925] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 988.021320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.067500] alg: No test for adler32 (adler32-zlib) [ 988.929702] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 989.027178] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 990.607155] Key type lgssc registered [ 990.949300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.981686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.988837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.072342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 993.078732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 993.132648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.463420] Lustre: Failing over lustre-MDT0000 [ 993.567122] Lustre: server umount lustre-MDT0000 complete [ 995.679843] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:57:29 (1782928649) [ 997.884983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.029746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.322899] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1002.220746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.258724] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1002.270427] Lustre: 64497:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1002.377723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.771916] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1006.291730] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.134330] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1010.299472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.370136] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1010.440829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.345937] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1014.651756] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1016.329327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1022.825799] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1026.529583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.530528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.532788] Lustre: Skipped 2 previous similar messages [ 1026.534260] Lustre: Skipped 1 previous similar message [ 1030.432219] Lustre: server umount lustre-MDT0000 complete [ 1031.742661] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928685 with bad export cookie 14124269339441710164 [ 1031.743974] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.747355] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1031.865289] Lustre: server umount lustre-MDT0001 complete [ 1033.066594] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1034.668985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1037.584286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.887733] LustreError: 63418:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008813dd66 x1869536183746560/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1042.001786] LustreError: 66342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.012610] LustreError: 66342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1042.017617] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.037820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.062996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1043.383560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1046.277237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.871710] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1050.348152] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.621137] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1052.187281] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1052.640090] Lustre: 63421:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782928690/real 1782928690] req@00000000a9961f0d x1869536183746304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782928706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1056.738050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.738301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.738561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.738566] Lustre: Skipped 1 previous similar message [ 1056.741823] Lustre: Skipped 1 previous similar message [ 1057.759241] Lustre: 63422:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782928695/real 1782928695] req@000000006414b459 x1869536183746432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782928711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1057.760856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.773518] Lustre: Skipped 3 previous similar messages [ 1060.061734] Lustre: server umount lustre-MDT0000 complete [ 1061.375301] LustreError: 67073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928715 with bad export cookie 14124269339441712432 [ 1061.378406] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.378917] LustreError: 67073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1061.505181] Lustre: server umount lustre-MDT0001 complete [ 1063.663195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1066.857086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.031738] LustreError: 63418:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f0cf7605 x1869536183765504/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1072.181756] LustreError: 68286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.187785] LustreError: 68286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1072.212622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.215836] Lustre: Skipped 1 previous similar message [ 1072.237449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1073.518177] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1076.294360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.390415] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.697710] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1078.495086] Lustre: 63419:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782928716/real 1782928716] req@000000002a58f3a0 x1869536183765120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782928732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1079.877764] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1081.732344] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1081.828302] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1083.359062] Lustre: 63420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782928720/real 1782928720] req@00000000b0441eec x1869536183765248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782928736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1086.947831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1086.948427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.949918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.957303] Lustre: Skipped 1 previous similar message [ 1087.967179] Lustre: 63422:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782928725/real 1782928725] req@00000000db1608e6 x1869536183765632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782928741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1087.976315] Lustre: 63422:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1089.566214] Lustre: server umount lustre-MDT0000 complete [ 1090.831243] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928744 with bad export cookie 14124269339441713804 [ 1090.835370] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.836309] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1090.955911] Lustre: server umount lustre-MDT0001 complete [ 1093.227958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1094.406481] Lustre: DEBUG MARKER: files compared the same [ 1104.430378] Lustre: server umount lustre-OST0000 complete [ 1110.139952] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1111.227323] Key type lgssc unregistered [ 1111.356406] LNet: 74212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.359948] LNetError: 74212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.373322] LNet: Removed LNI 192.168.204.144@tcp [ 1111.677484] Key type .llcrypt unregistered [ 1111.678817] Key type ._llcrypt unregistered [ 1119.907993] Key type ._llcrypt registered [ 1119.909340] Key type .llcrypt registered [ 1119.955133] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1120.337488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.396854] alg: No test for adler32 (adler32-zlib) [ 1121.256393] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1121.344581] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1122.927133] Key type lgssc registered [ 1123.271829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1125.824474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1125.830192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.973489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.250274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1131.057595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.163947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1132.532918] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1134.983379] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.707797] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1138.878849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1138.975355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.836411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1143.328391] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1153.313829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1156.575495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1156.579541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.584031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.623651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.624080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.624391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.626984] Lustre: Skipped 1 previous similar message [ 1162.253260] Lustre: server umount lustre-OST0000 complete [ 1163.743638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.743984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.749641] Lustre: Skipped 1 previous similar message [ 1163.753795] Lustre: Skipped 1 previous similar message [ 1168.863451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.865987] Lustre: Skipped 1 previous similar message [ 1169.620897] Lustre: server umount lustre-MDT0000 complete [ 1170.836884] LustreError: 75586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928824 with bad export cookie 5254192459664567346 [ 1170.838784] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.841517] LustreError: 75586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1170.945305] Lustre: server umount lustre-MDT0001 complete [ 1174.590249] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1175.593816] Key type lgssc unregistered [ 1175.733350] LNet: 79152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.735897] LNetError: 79152:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.745314] LNet: Removed LNI 192.168.204.144@tcp [ 1176.039682] Key type .llcrypt unregistered [ 1176.040680] Key type ._llcrypt unregistered [ 1184.861657] Key type ._llcrypt registered [ 1184.862730] Key type .llcrypt registered [ 1184.904780] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1185.385787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.397516] alg: No test for adler32 (adler32-zlib) [ 1186.256871] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1186.362149] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1187.951153] Key type lgssc registered [ 1188.312760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1190.968731] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.978967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.431359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1211.551293] LustreError: 80519:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 1238.175195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.090765] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1242.514665] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1245.008122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.048215] Lustre: *** cfs_fail_loc=706, val=0*** [ 1245.189964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.222224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1246.564048] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1249.369575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.490507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.794634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1253.273516] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1255.033731] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.956985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.958149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.960935] Lustre: Skipped 1 previous similar message [ 1263.075369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.077204] Lustre: Skipped 1 previous similar message [ 1268.194268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.196642] Lustre: Skipped 1 previous similar message [ 1270.239143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1270.288566] Lustre: server umount lustre-OST0000 complete [ 1273.312700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.313569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.316018] Lustre: Skipped 1 previous similar message [ 1273.320161] Lustre: Skipped 2 previous similar messages [ 1277.661364] Lustre: server umount lustre-MDT0000 complete [ 1278.432199] LustreError: 83394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.441303] LustreError: 83394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1278.915516] LustreError: 81645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782928932 with bad export cookie 13872618822876244158 [ 1278.919937] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.920184] LustreError: 81645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1279.045500] Lustre: server umount lustre-MDT0001 complete [ 1282.541528] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1283.524905] Key type lgssc unregistered [ 1283.645273] LNet: 84145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.647932] LNetError: 84145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.654292] LNet: Removed LNI 192.168.204.144@tcp [ 1283.956040] Key type .llcrypt unregistered [ 1283.958063] Key type ._llcrypt unregistered [ 1293.016557] Key type ._llcrypt registered [ 1293.017688] Key type .llcrypt registered [ 1293.054863] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1293.353519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1293.366207] alg: No test for adler32 (adler32-zlib) [ 1294.221715] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1294.319828] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1295.895141] Key type lgssc registered [ 1296.238788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.750873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.756714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.071513] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1303.586407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.758666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1305.127025] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1307.952055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1308.062963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.933912] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1312.652098] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1312.657120] Lustre: MGS: Client 19f57688-c8a9-415c-8767-5b07977c0f26 (at 0@lo) reconnecting [ 1312.743458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.774923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1314.102895] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1322.976294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1322.979541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.983251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1324.002604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.003474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1324.006039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1324.007504] Lustre: Skipped 1 previous similar message [ 1327.953152] Lustre: server umount lustre-OST0000 complete [ 1334.241074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.241168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.245596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.250604] Lustre: Skipped 1 previous similar message [ 1335.298956] Lustre: server umount lustre-MDT0000 complete [ 1336.634219] Lustre: server umount lustre-MDT0001 complete [ 1337.920773] Lustre: server umount MGS complete [ 1341.861512] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:03:15 (1782928995) [ 1351.400263] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1354.854784] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1366.497655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.810100] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1371.638064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.733134] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1371.772387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1371.819849] Lustre: lustre-MDT0001: new disk, initializing [ 1371.846410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.854588] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1371.856472] Lustre: 93383:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1371.858634] LustreError: 93383:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1373.133183] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1377.434332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.569540] Lustre: lustre-OST0000: new disk, initializing [ 1377.572391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1377.576312] Lustre: 94150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1377.608224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.628749] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1380.639799] Lustre: 94488:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000038262480 x1869536503563648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782929050 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1383.355641] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1385.524509] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.528354] Lustre: MGS: Client df2bd248-95f1-4cad-8766-81c95a9bb509 (at 0@lo) reconnecting [ 1385.542995] Lustre: 95197:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.549461] LustreError: 93369:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1385.559412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1385.595987] Lustre: lustre-MDT0000: new disk, initializing [ 1385.637023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.650084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1386.719841] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1386.720062] Lustre: 95221:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b69edc28 x1869536503569792/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782929056 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1386.729791] Lustre: 95221:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 1386.956629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1395.169752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1395.172463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1395.810468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1395.811598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1416.673219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.679052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.688923] Lustre: Skipped 1 previous similar message [ 1416.692128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.167129] Lustre: server umount lustre-OST0000 complete [ 1428.448605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.448935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.452086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.455267] Lustre: Skipped 2 previous similar messages [ 1428.457326] Lustre: Skipped 2 previous similar messages [ 1429.635962] Lustre: server umount lustre-MDT0000 complete [ 1445.343131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.428092] Lustre: server umount lustre-MDT0001 complete [ 1446.701158] Lustre: server umount MGS complete [ 1450.240375] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:05:03 (1782929103) [ 1456.028974] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1457.103110] Key type lgssc unregistered [ 1457.236333] LNet: 97718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.238617] LNetError: 97718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.250287] LNet: Removed LNI 192.168.204.144@tcp [ 1457.598770] Key type .llcrypt unregistered [ 1457.600560] Key type ._llcrypt unregistered [ 1464.614410] Key type ._llcrypt registered [ 1464.615433] Key type .llcrypt registered [ 1464.665796] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1465.144916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.220310] alg: No test for adler32 (adler32-zlib) [ 1466.081455] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1466.175088] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1467.759211] Key type lgssc registered [ 1468.148315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.700512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.707801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.850495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.227803] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1476.148984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.264943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.585746] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1480.001077] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1481.868983] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1484.107704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.218158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.154562] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1488.579704] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1496.486738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1506.785168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.786702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.788231] LustreError: Skipped 1 previous similar message [ 1506.789147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1506.792683] Lustre: Skipped 1 previous similar message [ 1506.798235] Lustre: Skipped 1 previous similar message [ 1507.991928] Lustre: server umount lustre-OST0000 complete [ 1511.905682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.906762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.910770] Lustre: Skipped 1 previous similar message [ 1515.416854] Lustre: server umount lustre-MDT0000 complete [ 1516.612600] LustreError: 98897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929170 with bad export cookie 4421857485233403354 [ 1516.617768] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1516.618349] LustreError: 98897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1516.748304] Lustre: server umount lustre-MDT0001 complete [ 1520.384325] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1521.474731] Key type lgssc unregistered [ 1521.612490] LNet: 102465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.614966] LNetError: 102465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.622306] LNet: Removed LNI 192.168.204.144@tcp [ 1521.929613] Key type .llcrypt unregistered [ 1521.931257] Key type ._llcrypt unregistered [ 1528.535273] Key type ._llcrypt registered [ 1528.536393] Key type .llcrypt registered [ 1528.573923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1528.888770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1528.894599] alg: No test for adler32 (adler32-zlib) [ 1529.758350] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1529.851780] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1531.431143] Key type lgssc registered [ 1531.803625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1533.932659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1533.932758] LustreError: 103526:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1533.936448] Lustre: Skipped 1 previous similar message [ 1533.940677] LustreError: 103525:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1533.944432] LustreError: 103526:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1533.947487] LustreError: 103526:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1533.951754] LustreError: 103526:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1533.952467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.137113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.734159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.844688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1539.185336] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1540.818971] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lsmod [ 1541.755379] LustreError: 104728:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1541.755411] LustreError: 104729:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1541.761896] LustreError: 104728:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1541.764311] LustreError: 104728:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1541.767845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1541.769185] LustreError: 104728:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1541.886634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.844842] LustreError: 103591:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1542.847796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.848439] LustreError: 103591:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1542.849281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.851745] Lustre: Skipped 2 previous similar messages [ 1542.859108] LustreError: 103591:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1548.001389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.004888] Lustre: Skipped 4 previous similar messages [ 1549.012514] Lustre: server umount lustre-MDT0000 complete [ 1550.241231] LustreError: 103541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929203 with bad export cookie 9288968630624389112 [ 1550.245485] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1550.246131] LustreError: 103541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1552.095248] LustreError: 103992:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c6fa2c84 x1869536751008256/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1552.154864] Lustre: server umount lustre-MDT0001 complete [ 1563.674856] Lustre: server umount lustre-OST0000 complete [ 1566.536885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.690328] LustreError: 105796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.696815] LustreError: 105796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1566.713031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.088586] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1570.955941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.079443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.363893] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1574.633165] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1576.452560] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1578.611210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.531959] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1582.834285] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1593.062785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1596.383881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.386336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.390507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.432996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.434263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.435717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.856646] Lustre: server umount lustre-OST0000 complete [ 1603.552744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.553484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.557110] Lustre: Skipped 1 previous similar message [ 1603.559286] Lustre: Skipped 1 previous similar message [ 1608.347274] Lustre: server umount lustre-MDT0000 complete [ 1608.671985] LustreError: 105796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.676494] LustreError: 105796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1609.560204] LustreError: 105777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929263 with bad export cookie 9288968630624389777 [ 1609.563386] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.563530] LustreError: 105777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1609.675947] Lustre: server umount lustre-MDT0001 complete [ 1613.155896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1614.140883] Key type lgssc unregistered [ 1614.266346] LNet: 109349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.268995] LNetError: 109349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.278254] LNet: Removed LNI 192.168.204.144@tcp [ 1614.570483] Key type .llcrypt unregistered [ 1614.571798] Key type ._llcrypt unregistered [ 1623.036961] Key type ._llcrypt registered [ 1623.039207] Key type .llcrypt registered [ 1623.080698] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1623.348911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1623.363615] alg: No test for adler32 (adler32-zlib) [ 1624.213810] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1624.296580] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1625.879116] Key type lgssc registered [ 1626.226952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.779293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.784611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.922621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.236757] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1634.219735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.352944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.617788] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1638.064528] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1639.901033] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1642.278059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.409932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.367253] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1646.676113] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1655.718824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1659.085202] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1659.937428] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1663.967965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.970893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.976491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.017732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.018933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.025822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1671.137249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1671.138992] Lustre: Skipped 2 previous similar messages [ 1675.231100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.278039] Lustre: server umount lustre-OST0000 complete [ 1680.351742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1680.353871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.357330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.582961] Lustre: server umount lustre-MDT0000 complete [ 1683.782463] LustreError: 110724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929337 with bad export cookie 14997817875468119132 [ 1683.786283] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.788217] LustreError: 110724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1683.902453] Lustre: server umount lustre-MDT0001 complete [ 1687.518652] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1688.484583] Key type lgssc unregistered [ 1688.612314] LNet: 114396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.614982] LNetError: 114396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.624290] LNet: Removed LNI 192.168.204.144@tcp [ 1688.909066] Key type .llcrypt unregistered [ 1688.910054] Key type ._llcrypt unregistered [ 1695.704560] Key type ._llcrypt registered [ 1695.705800] Key type .llcrypt registered [ 1695.740225] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1696.047656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.092162] alg: No test for adler32 (adler32-zlib) [ 1696.943832] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1697.040714] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1698.623130] Key type lgssc registered [ 1699.028136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1701.541845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1701.549157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.687907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.966254] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1706.724969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.840423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1708.130201] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1710.476303] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1712.299850] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1714.420446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.525103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.539081] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1718.729323] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1727.846490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1732.063893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.066287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.070172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.089096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.090140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.090925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.876100] Lustre: server umount lustre-OST0000 complete [ 1742.815892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1742.819832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.823896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.825560] Lustre: Skipped 1 previous similar message [ 1745.174730] Lustre: server umount lustre-MDT0000 complete [ 1746.399222] LustreError: 115577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929400 with bad export cookie 12882374624418966443 [ 1746.403175] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1746.404139] LustreError: 115577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1746.514870] Lustre: server umount lustre-MDT0001 complete [ 1750.060309] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1751.052917] Key type lgssc unregistered [ 1751.189305] LNet: 119148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.192118] LNetError: 119148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.200258] LNet: Removed LNI 192.168.204.144@tcp [ 1751.516698] Key type .llcrypt unregistered [ 1751.518236] Key type ._llcrypt unregistered [ 1760.650727] Key type ._llcrypt registered [ 1760.651893] Key type .llcrypt registered [ 1760.698304] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1761.017020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.065766] alg: No test for adler32 (adler32-zlib) [ 1761.910050] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1761.994311] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1763.567104] Key type lgssc registered [ 1763.922223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1766.419163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1766.427134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.565835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.808823] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1771.585702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.707674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1773.064824] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1775.690807] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1778.249865] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1782.118978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1782.245020] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1782.306397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.540985] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1789.914950] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1792.696185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1798.124363] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1799.988813] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1799.992692] Lustre: Skipped 2 previous similar messages [ 1802.164764] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1802.169515] Lustre: Skipped 2 previous similar messages [ 1807.377277] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1807.391577] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1811.104246] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1811.113716] Lustre: Skipped 2 previous similar messages [ 1816.060294] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 1818.486255] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.44/24@tcp in log lustre-MDT0000 [ 1818.491140] Lustre: Skipped 2 previous similar messages [ 1822.718399] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 1822.720060] Lustre: Skipped 1 previous similar message [ 1828.833418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.834934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.837286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.840870] Lustre: Skipped 1 previous similar message [ 1830.164413] Lustre: server umount lustre-OST0000 complete [ 1833.952925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1833.953478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.955827] LustreError: Skipped 1 previous similar message [ 1833.962173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.966242] Lustre: Skipped 2 previous similar messages [ 1837.658207] Lustre: server umount lustre-MDT0000 complete [ 1838.980300] LustreError: 120520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929492 with bad export cookie 16067146620762244314 [ 1838.983898] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.989717] LustreError: 120520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1839.104306] Lustre: server umount lustre-MDT0001 complete [ 1843.454090] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1844.691180] Key type lgssc unregistered [ 1844.844623] LNet: 125612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.847324] LNetError: 125612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.855574] LNet: Removed LNI 192.168.204.144@tcp [ 1845.271366] Key type .llcrypt unregistered [ 1845.272600] Key type ._llcrypt unregistered [ 1857.272052] Key type ._llcrypt registered [ 1857.273532] Key type .llcrypt registered [ 1857.316904] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1857.694569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.748530] alg: No test for adler32 (adler32-zlib) [ 1858.629710] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1858.729408] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1860.327159] Key type lgssc registered [ 1860.781713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.518372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.527116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.690790] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1864.693878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.701036] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.44/24@tcp [ 1866.170074] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1869.138977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.243345] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1869.252747] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 1869.269429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.703954] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1873.230187] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1875.054252] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.271958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.406164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.243755] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1881.750078] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.703202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1898.976059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1898.979821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.986447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.001796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.003686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.003703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.797649] Lustre: server umount lustre-OST0000 complete [ 1905.120723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.121058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.121756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.121772] Lustre: Skipped 1 previous similar message [ 1905.131847] Lustre: Skipped 1 previous similar message [ 1910.242140] LustreError: 127890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.251328] LustreError: 127890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1910.253263] Lustre: server umount lustre-MDT0000 complete [ 1911.767517] LustreError: 127137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929565 with bad export cookie 18435349401695689657 [ 1911.768183] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.772890] LustreError: 127137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.921766] Lustre: server umount lustre-MDT0001 complete [ 1916.013991] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1917.280965] Key type lgssc unregistered [ 1917.422402] LNet: 130768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.426832] LNetError: 130768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.439445] LNet: Removed LNI 192.168.204.144@tcp [ 1917.804300] Key type .llcrypt unregistered [ 1917.806409] Key type ._llcrypt unregistered [ 1927.873638] Key type ._llcrypt registered [ 1927.876193] Key type .llcrypt registered [ 1927.918571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1934.440663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1934.830939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.878623] alg: No test for adler32 (adler32-zlib) [ 1935.776260] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 1935.908111] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1937.511144] Key type lgssc registered [ 1938.005105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.909170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1950.917216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.006403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1952.018041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1952.059117] Lustre: lustre-MDT0000: new disk, initializing [ 1952.081861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.088064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1953.625433] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1958.451868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.504571] Lustre: 134140:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1958.550535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1958.553094] Lustre: Skipped 1 previous similar message [ 1958.596343] Lustre: lustre-MDT0001: new disk, initializing [ 1958.654280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.673365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1958.679783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1960.413206] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1964.838395] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1966.957703] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1969.805074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.928466] Lustre: lustre-OST0000: new disk, initializing [ 1969.931065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1969.935040] Lustre: 136110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1969.972562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.487534] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1973.190596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1973.195219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1973.242329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1976.825396] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1978.433267] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1978.528644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1980.176543] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1980.270286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1983.457046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.457398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.460064] LustreError: Skipped 1 previous similar message [ 1983.465665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.470173] Lustre: Skipped 1 previous similar message [ 1988.577793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.579862] Lustre: Skipped 1 previous similar message [ 1993.699116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.702153] Lustre: Skipped 1 previous similar message [ 1995.231151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1995.291582] Lustre: server umount lustre-OST0000 complete [ 1998.817594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.818272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.822405] Lustre: Skipped 2 previous similar messages [ 2002.834196] Lustre: server umount lustre-MDT0000 complete [ 2004.055231] LustreError: 134134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929657 with bad export cookie 2915732341544158800 [ 2004.059378] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.059484] LustreError: 134134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2004.175654] Lustre: server umount lustre-MDT0001 complete [ 2007.227678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.404195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.795987] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2011.834686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.965641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.398354] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2015.957369] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2017.875423] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.342805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.480193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.528130] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2025.219113] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2026.847410] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2033.101348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2034.881080] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2034.975248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2039.760028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2039.843722] Lustre: lustre-OST0001: new disk, initializing [ 2039.845797] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2039.849659] Lustre: 141741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2039.895581] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.488236] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2045.971200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2045.976203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2046.025544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2046.651874] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2046.766312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2048.734640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2048.872841] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2051.759549] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f05c5525000.ost_server_uuid 50 [ 2052.465756] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f05c5525000.ost_server_uuid in FULL state after 0 sec [ 2056.165299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.167221] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.173657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2056.176203] Lustre: Skipped 2 previous similar messages [ 2061.593270] Lustre: server umount lustre-OST0001 complete [ 2063.840661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2063.845611] LustreError: Skipped 1 previous similar message [ 2063.847628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.854984] Lustre: Skipped 1 previous similar message [ 2066.403127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.403834] LustreError: 141520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.406900] Lustre: Skipped 3 previous similar messages [ 2066.423773] LustreError: 141520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2069.677475] Lustre: server umount lustre-OST0000 complete [ 2073.568138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2073.572937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.579619] Lustre: Skipped 1 previous similar message [ 2077.739498] Lustre: server umount lustre-MDT0000 complete [ 2079.677608] LustreError: 138308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929733 with bad export cookie 2915732341544159682 [ 2079.682422] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.688997] LustreError: 138308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2079.836310] Lustre: server umount lustre-MDT0001 complete [ 2088.068241] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2089.311858] Key type lgssc unregistered [ 2089.447583] LNet: 144737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.450931] LNetError: 144737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.460454] LNet: Removed LNI 192.168.204.144@tcp [ 2089.873332] Key type .llcrypt unregistered [ 2089.874573] Key type ._llcrypt unregistered [ 2108.835776] Key type ._llcrypt registered [ 2108.837821] Key type .llcrypt registered [ 2108.885749] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2116.657335] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2117.194237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.214172] alg: No test for adler32 (adler32-zlib) [ 2118.118218] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 2118.257473] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2119.863177] Key type lgssc registered [ 2120.420220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.755637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2135.766490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.858967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2136.869646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2136.907746] Lustre: lustre-MDT0000: new disk, initializing [ 2136.939190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.947461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2138.545615] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2143.542323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.592976] Lustre: 148861:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2143.623018] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2143.624755] Lustre: Skipped 1 previous similar message [ 2143.664070] Lustre: lustre-MDT0001: new disk, initializing [ 2143.704476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.721893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2143.726224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2145.523381] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2150.397350] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.705754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2155.601853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.719850] Lustre: lustre-OST0000: new disk, initializing [ 2155.721861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2155.725397] Lustre: 150829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2155.763524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.124582] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2162.007088] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2165.765157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2165.769435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2165.813070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2169.492282] Lustre: Failing over lustre-OST0000 [ 2169.563420] Lustre: server umount lustre-OST0000 complete [ 2183.191735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.340496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.345673] Lustre: Skipped 1 previous similar message [ 2183.352164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.366695] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2183.853029] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2185.921159] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2186.399322] Lustre: 147093:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782929824/real 1782929824] req@000000007f1473ea x1869537367585024/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782929840 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.232067] Lustre: 147091:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782929824/real 1782929824] req@000000008a2cc01e x1869537367584768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782929840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.243289] Lustre: 147091:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2187.556647] Lustre: Failing over lustre-MDT0000 [ 2187.677425] Lustre: server umount lustre-MDT0000 complete [ 2188.768918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.773722] Lustre: Skipped 2 previous similar messages [ 2191.839153] Lustre: 147090:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782929829/real 1782929829] req@0000000087f7772d x1869537367586176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782929845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.851549] Lustre: 147090:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2193.888464] LustreError: 150210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.895920] LustreError: 150210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2196.895159] Lustre: 147091:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782929834/real 1782929834] req@00000000e3a945cc x1869537367587328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782929850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.009618] LustreError: 150210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.016557] LustreError: 150210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2201.562164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.629298] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.636599] Lustre: 148866:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2201.723276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2201.748280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.781420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2203.417560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2204.320596] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2207.205921] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2207.216114] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2207.250939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2212.322628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.324983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.326044] LustreError: Skipped 1 previous similar message [ 2212.332523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.337897] Lustre: Skipped 1 previous similar message [ 2214.744075] Lustre: server umount lustre-OST0000 complete [ 2217.440939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.441173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.441700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.447036] Lustre: Skipped 2 previous similar messages [ 2222.310474] Lustre: server umount lustre-MDT0000 complete [ 2222.559958] LustreError: 149671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.568055] LustreError: 149671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2223.929297] LustreError: 148855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782929877 with bad export cookie 1799687986206109661 [ 2223.932368] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.934675] LustreError: 148855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2224.061410] Lustre: server umount lustre-MDT0001 complete [ 2228.481676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2229.748882] Key type lgssc unregistered [ 2229.900114] LNet: 154477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.906527] LNetError: 154477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.920829] LNet: Removed LNI 192.168.204.144@tcp [ 2230.293418] Key type .llcrypt unregistered [ 2230.294821] Key type ._llcrypt unregistered [ 2252.187571] Key type ._llcrypt registered [ 2252.189073] Key type .llcrypt registered [ 2252.231596] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2252.577946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.605625] alg: No test for adler32 (adler32-zlib) [ 2253.478589] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 2253.575097] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2255.159341] Key type lgssc registered [ 2255.634144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.688082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.697692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.874381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.393162] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2265.444260] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2267.583632] LustreError: 156024:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2267.588681] LustreError: 156024:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2267.592205] LustreError: 156024:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2267.716804] Lustre: server umount lustre-MDT0000 complete [ 2273.683877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2277.733470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2291.428088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.520224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2291.533316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2291.570907] Lustre: lustre-MDT0000: new disk, initializing [ 2291.599181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.606222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2293.130509] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2297.930889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.981276] Lustre: 159322:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2298.010516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2298.013061] Lustre: Skipped 1 previous similar message [ 2298.050773] Lustre: lustre-MDT0001: new disk, initializing [ 2298.093269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2298.108857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2298.112708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2299.717552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2303.956396] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2306.234546] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2308.971678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.098250] Lustre: lustre-OST0000: new disk, initializing [ 2309.100460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2309.104748] Lustre: 161289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2309.143502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.405411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2310.411661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2310.456369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2311.536076] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2315.774706] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2318.865970] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2319.565350] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:19:32 (1782929972) [ 2407.903905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.904230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.909387] Lustre: Skipped 2 previous similar messages [ 2407.912097] Lustre: Skipped 1 previous similar message [ 2413.025062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.028554] Lustre: Skipped 3 previous similar messages [ 2418.143606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.146424] Lustre: Skipped 2 previous similar messages [ 2421.215117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.285183] Lustre: server umount lustre-MDT0000 complete [ 2422.616523] LustreError: 160118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930076 with bad export cookie 2167481005531877735 [ 2422.617750] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2422.621765] LustreError: 160118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2423.263439] LustreError: 159329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.263509] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.269666] LustreError: 159329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2423.279729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2427.295442] LustreError: 159329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.295442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2428.776337] Lustre: server umount lustre-MDT0001 complete [ 2436.303084] Lustre: server umount lustre-OST0000 complete [ 2439.859023] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2443.634504] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2456.754698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.837563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2456.848306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2456.887917] Lustre: lustre-MDT0000: new disk, initializing [ 2456.912063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.919587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2458.373875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2463.159104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.217088] Lustre: 165916:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2463.253172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2463.256131] Lustre: Skipped 1 previous similar message [ 2463.295870] Lustre: lustre-MDT0001: new disk, initializing [ 2463.361770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2463.367715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2464.920401] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2469.065628] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2471.078806] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2473.652426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2473.758308] Lustre: lustre-OST0000: new disk, initializing [ 2473.760371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2473.763265] Lustre: 167882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2473.793200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.796149] Lustre: Skipped 1 previous similar message [ 2475.931814] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2476.088398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2476.092983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2476.133462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2479.753311] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2481.132559] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2481.202478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2482.555858] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2482.629952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2486.242579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2486.244204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.245781] LustreError: Skipped 1 previous similar message [ 2486.246710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.250899] Lustre: Skipped 1 previous similar message [ 2489.429859] Lustre: server umount lustre-OST0000 complete [ 2491.361245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.365469] Lustre: Skipped 1 previous similar message [ 2505.183109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2505.246377] Lustre: server umount lustre-MDT0000 complete [ 2506.566125] LustreError: 165908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930160 with bad export cookie 2167481005532036565 [ 2506.569100] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.570793] LustreError: 165908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2506.693935] Lustre: server umount lustre-MDT0001 complete [ 2509.797454] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:22:43 (1782930163) [ 2515.159976] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2518.392322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2530.940841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.045838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2531.061593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2531.102969] Lustre: lustre-MDT0000: new disk, initializing [ 2531.135733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.145205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2532.686272] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2536.866021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.911163] Lustre: 172631:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2537.011316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2538.403743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2542.216528] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.061549] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2546.427083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2546.537234] Lustre: 174610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2548.511575] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2552.368299] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2555.390185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2555.431031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2560.481918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2560.482063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2560.484681] LustreError: Skipped 1 previous similar message [ 2560.488931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2560.491078] Lustre: Skipped 8 previous similar messages [ 2561.628251] Lustre: server umount lustre-OST0000 complete [ 2569.048787] Lustre: server umount lustre-MDT0000 complete [ 2570.258068] LustreError: 173433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930223 with bad export cookie 2167481005532037433 [ 2570.261578] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2570.263187] LustreError: 173433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2570.369717] Lustre: server umount lustre-MDT0001 complete [ 2573.091801] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:23:46 (1782930226) [ 2578.160707] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2581.099320] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2591.946664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.027207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2592.036424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2592.037808] Lustre: Skipped 3 previous similar messages [ 2592.069835] Lustre: lustre-MDT0000: new disk, initializing [ 2592.071211] Lustre: Skipped 2 previous similar messages [ 2592.089449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.091533] Lustre: Skipped 2 previous similar messages [ 2592.096196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2592.099068] Lustre: Skipped 2 previous similar messages [ 2593.334091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2597.273117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.317240] Lustre: 178926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2597.351638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2597.353286] Lustre: Skipped 2 previous similar messages [ 2597.431237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2598.726154] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2602.244607] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2603.911309] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2605.977402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.073896] Lustre: 180901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2607.069669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2607.965851] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2611.508321] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2617.313430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2617.314348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.320632] Lustre: Skipped 4 previous similar messages [ 2619.284253] Lustre: server umount lustre-OST0000 complete [ 2626.774364] Lustre: server umount lustre-MDT0000 complete [ 2627.552113] LustreError: 181173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.558837] LustreError: 181173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2628.052145] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930281 with bad export cookie 2167481005532038504 [ 2628.054055] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.057656] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2628.170731] Lustre: server umount lustre-MDT0001 complete [ 2631.835433] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2632.820666] Key type lgssc unregistered [ 2632.946297] LNet: 182756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.948744] LNetError: 182756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.955323] LNet: Removed LNI 192.168.204.144@tcp [ 2633.261858] Key type .llcrypt unregistered [ 2633.263116] Key type ._llcrypt unregistered [ 2642.397446] Key type ._llcrypt registered [ 2642.398772] Key type .llcrypt registered [ 2642.437613] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2642.839600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.868522] alg: No test for adler32 (adler32-zlib) [ 2643.718985] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 2643.803475] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2645.383136] Key type lgssc registered [ 2645.722139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.088786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.096248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.228055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.494439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2653.234207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.355902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.737519] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2657.004934] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.783025] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2661.078117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.215986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.142463] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2665.417772] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2673.510020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2688.992836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.994474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.995572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.998348] Lustre: Skipped 1 previous similar message [ 2689.001883] Lustre: Skipped 1 previous similar message [ 2693.525332] Lustre: server umount lustre-OST0000 complete [ 2699.232518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.233235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.237896] Lustre: Skipped 1 previous similar message [ 2700.885418] Lustre: server umount lustre-MDT0000 complete [ 2702.098440] LustreError: 184129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930355 with bad export cookie 4014124171410136724 [ 2702.101595] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.102834] LustreError: 184129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2702.225149] Lustre: server umount lustre-MDT0001 complete [ 2705.807398] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2706.915308] Key type lgssc unregistered [ 2707.055362] LNet: 187703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.057963] LNetError: 187703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.066345] LNet: Removed LNI 192.168.204.144@tcp [ 2707.394939] Key type .llcrypt unregistered [ 2707.397238] Key type ._llcrypt unregistered [ 2716.764881] Key type ._llcrypt registered [ 2716.766581] Key type .llcrypt registered [ 2716.801181] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2717.110919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2717.140317] alg: No test for adler32 (adler32-zlib) [ 2717.985903] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 2718.070926] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2719.647162] Key type lgssc registered [ 2720.009529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.550891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2722.557986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.693909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.993123] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2727.809095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.935362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.218960] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2731.398812] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.127695] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2735.238392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.339331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.052101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2739.215835] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2748.644075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2752.991781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.993956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.997649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2754.016537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2754.017258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.017609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.137748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.139917] Lustre: Skipped 3 previous similar messages [ 2764.258076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.260168] Lustre: Skipped 2 previous similar messages [ 2766.303106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.356901] Lustre: server umount lustre-OST0000 complete [ 2768.741820] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f05c79b3800.ost_server_uuid 50 [ 2769.246986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f05c79b3800.ost_server_uuid in DISCONN state after 0 sec [ 2769.792865] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2784.223845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.226038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.229706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.231184] Lustre: Skipped 1 previous similar message [ 2785.875868] Lustre: server umount lustre-MDT0000 complete [ 2787.026050] LustreError: 189078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930440 with bad export cookie 10192280273379390899 [ 2787.029098] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.031369] LustreError: 189078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2787.134875] Lustre: server umount lustre-MDT0001 complete [ 2789.832619] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:27:23 (1782930443) [ 2792.010688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.162024] LustreError: 193053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.167778] LustreError: 193053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2792.182713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.468419] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2796.177222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.304200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.654429] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2799.934016] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2801.694604] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2803.819808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.592944] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2807.676142] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2809.805216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.885725] Lustre: lustre-OST0001: new disk, initializing [ 2809.886980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2809.888827] Lustre: 195890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2809.922258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.924665] Lustre: Skipped 1 previous similar message [ 2810.172645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2810.175827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2810.203228] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2811.796864] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2815.417291] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2816.479905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.483259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.488443] Lustre: Skipped 1 previous similar message [ 2816.490808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2816.493012] Lustre: Skipped 2 previous similar messages [ 2822.331049] Lustre: server umount lustre-OST0000 complete [ 2822.625952] LustreError: 194873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.630735] LustreError: 194873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2824.217894] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2824.290393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2825.559103] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2825.628142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2825.886505] LustreError: 195305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.005970] LustreError: 195945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.010925] LustreError: 195945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2835.936051] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2835.936286] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.939019] LustreError: Skipped 1 previous similar message [ 2835.942800] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2835.945931] Lustre: Skipped 4 previous similar messages [ 2837.984952] LustreError: 195305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.991619] LustreError: 195305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2841.817319] Lustre: server umount lustre-OST0001 complete [ 2849.244044] Lustre: server umount lustre-MDT0000 complete [ 2850.528065] LustreError: 193035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930504 with bad export cookie 10192280273379391956 [ 2850.531373] LustreError: 193035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2850.532449] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.648239] Lustre: server umount lustre-MDT0001 complete [ 2858.805928] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:28:32 (1782930512) [ 2861.118128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.215031] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2861.227566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2861.298105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.614332] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2865.444847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.480549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2865.492224] Lustre: 199306:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2867.003360] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2869.268774] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2871.043698] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2873.153743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.223396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2875.039820] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2875.462942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2877.246325] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2879.377949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.402126] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2881.247884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2883.415077] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2883.480104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2885.599915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.600111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.602054] LustreError: Skipped 1 previous similar message [ 2885.606102] Lustre: Skipped 3 previous similar messages [ 2885.606568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.611010] Lustre: Skipped 7 previous similar messages [ 2890.235107] Lustre: server umount lustre-OST0000 complete [ 2893.795017] LustreError: 202853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.799190] LustreError: 202853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2904.033851] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.034631] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.035928] LustreError: Skipped 1 previous similar message [ 2904.040686] Lustre: Skipped 2 previous similar messages [ 2906.834498] Lustre: server umount lustre-OST0001 complete [ 2914.133932] Lustre: server umount lustre-MDT0000 complete [ 2914.273221] LustreError: 200498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.277554] LustreError: 200498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2915.285874] LustreError: 199343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930568 with bad export cookie 10192280273379393090 [ 2915.289378] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2915.289945] LustreError: 199343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2915.387418] Lustre: server umount lustre-MDT0001 complete [ 2922.836361] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:29:36 (1782930576) [ 2927.904628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2930.833519] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2941.625442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.703355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2941.714237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2941.749864] Lustre: lustre-MDT0000: new disk, initializing [ 2941.771846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.773566] Lustre: Skipped 3 previous similar messages [ 2941.779200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2943.035578] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2946.933027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.971313] Lustre: 207364:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2946.993543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2946.995243] Lustre: Skipped 1 previous similar message [ 2947.031418] Lustre: lustre-MDT0001: new disk, initializing [ 2947.073122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2947.077835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2948.269206] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2951.677103] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2953.259268] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2955.302050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.394542] Lustre: lustre-OST0000: new disk, initializing [ 2955.396107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2955.397970] Lustre: 209330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2957.160479] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2960.470968] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2961.687183] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2962.422345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2962.425697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2962.456643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2962.781163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2964.084516] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2964.154936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2967.521970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2967.522194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.524039] LustreError: Skipped 1 previous similar message [ 2967.524456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2967.528133] Lustre: Skipped 3 previous similar messages [ 2967.529471] Lustre: Skipped 7 previous similar messages [ 2970.899699] Lustre: server umount lustre-OST0000 complete [ 2978.400142] Lustre: server umount lustre-MDT0000 complete [ 2979.567135] LustreError: 208157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930633 with bad export cookie 10192280273379394854 [ 2979.568810] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.571038] LustreError: 208157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2979.688169] Lustre: server umount lustre-MDT0001 complete [ 2982.236354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.402695] LustreError: 211547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.720689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2986.429941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.921581] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2990.241536] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2991.900414] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2993.993300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.931246] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2998.227648] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.634471] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3007.848329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3009.084478] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3009.157574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3015.817128] Lustre: server umount lustre-OST0000 complete [ 3017.670951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3017.746454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3019.082920] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3042.472161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3042.588691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.591583] Lustre: Skipped 5 previous similar messages [ 3044.375905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3046.497868] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3046.979251] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3053.965222] Lustre: server umount lustre-OST0000 complete [ 3058.144761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3058.144822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.147591] LustreError: Skipped 3 previous similar messages [ 3058.150725] Lustre: Skipped 4 previous similar messages [ 3061.338683] Lustre: server umount lustre-MDT0000 complete [ 3062.605116] LustreError: 211529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930716 with bad export cookie 10192280273379395722 [ 3062.608057] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.609185] LustreError: 211529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3062.725840] Lustre: server umount lustre-MDT0001 complete [ 3065.458752] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:31:58 (1782930718) [ 3067.653379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.827842] LustreError: 217577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.257423] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3072.017802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.544136] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3075.953195] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3077.841103] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3080.175634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.331416] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3084.791926] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3086.325374] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3094.573956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3096.013767] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3096.100944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3098.416319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.477064] Lustre: lustre-OST0001: new disk, initializing [ 3098.478557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3098.480681] Lustre: 221001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3100.400815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3103.899954] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3105.148426] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3105.275281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3105.278289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3105.302936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3106.232183] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3107.489292] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3107.556838] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3110.374857] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3110.377457] Lustre: Skipped 11 previous similar messages [ 3114.237379] Lustre: server umount lustre-OST0001 complete [ 3116.057060] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3116.124736] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3117.367198] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3117.434641] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3141.015691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.077678] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3145.490570] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3146.033560] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3146.214740] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3146.218378] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3146.218576] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3146.223400] Lustre: Skipped 1 previous similar message [ 3152.761909] Lustre: server umount lustre-OST0001 complete [ 3170.904747] Lustre: server umount lustre-OST0000 complete [ 3178.394970] Lustre: server umount lustre-MDT0000 complete [ 3179.592215] LustreError: 219417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930833 with bad export cookie 10192280273379396653 [ 3179.595616] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.596672] LustreError: 219417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3185.775415] Lustre: server umount lustre-MDT0001 complete [ 3193.460848] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:34:06 (1782930846) [ 3195.570889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.659200] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3195.668451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3195.740527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.744404] Lustre: Skipped 5 previous similar messages [ 3197.021125] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3199.771325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.803947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3199.813915] Lustre: 226244:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3201.274657] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3203.584703] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3205.267455] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3207.380146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.442749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3209.310696] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3211.535565] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3220.560621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.583596] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3222.453260] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3224.766609] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3229.410233] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 3230.877395] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3230.953702] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3232.347791] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3232.418974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3234.272805] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f05d1727800.ost_server_uuid 50 [ 3234.803715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f05d1727800.ost_server_uuid in FULL state after 0 sec [ 3235.136038] Lustre: Permanently deactivating lustre-OST0001 [ 3235.140556] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3235.945982] Lustre: Permanently reactivating lustre-OST0001 [ 3238.372573] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.372808] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.377229] Lustre: Skipped 9 previous similar messages [ 3238.380901] LustreError: Skipped 7 previous similar messages [ 3243.003761] Lustre: server umount lustre-OST0001 complete [ 3243.495386] LustreError: 228764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.501960] LustreError: 228764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 3250.389899] Lustre: server umount lustre-OST0000 complete [ 3257.829309] Lustre: server umount lustre-MDT0000 complete [ 3259.113226] LustreError: 226236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930912 with bad export cookie 10192280273379397927 [ 3259.116335] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.116604] LustreError: 226236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3259.244043] Lustre: server umount lustre-MDT0001 complete [ 3267.050119] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:35:20 (1782930920) [ 3269.522402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.614048] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3269.624785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3269.628161] Lustre: Skipped 5 previous similar messages [ 3271.038966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3273.865937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.900717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3273.913236] Lustre: 232924:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3275.364844] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3277.765849] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3279.523730] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3281.767318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3281.852962] Lustre: Permanently deactivating lustre-OST0000 [ 3281.859198] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3283.814957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3284.401786] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3286.232322] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3288.900532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.098526] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3291.866188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3293.519349] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3294.088613] Lustre: Permanently reactivating lustre-OST0000 [ 3298.806414] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3298.833561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3310.331658] Lustre: server umount lustre-OST0001 complete [ 3317.790930] Lustre: server umount lustre-OST0000 complete [ 3325.207785] Lustre: server umount lustre-MDT0000 complete [ 3326.418280] LustreError: 232915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782930980 with bad export cookie 10192280273379399558 [ 3326.421846] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.422798] LustreError: 232915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3326.536827] Lustre: server umount lustre-MDT0001 complete [ 3330.067267] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3331.046837] Key type lgssc unregistered [ 3331.168325] LNet: 237772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.170420] LNetError: 237772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3331.176319] LNet: Removed LNI 192.168.204.144@tcp [ 3331.486273] Key type .llcrypt unregistered [ 3331.487300] Key type ._llcrypt unregistered [ 3339.686690] Key type ._llcrypt registered [ 3339.688298] Key type .llcrypt registered [ 3339.729492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3340.036754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3340.047548] alg: No test for adler32 (adler32-zlib) [ 3340.908802] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3341.002804] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3342.583173] Key type lgssc registered [ 3342.943760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.164844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.170177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.313318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.583435] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3351.309248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.350035] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3351.359667] Lustre: 239205:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3351.363448] Lustre: Permanently deactivating lustre-MDT0001 [ 3351.368229] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3351.476931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.822629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3353.067319] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3353.069709] LustreError: 239245:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3353.073024] LustreError: 239245:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3353.076101] LustreError: 239245:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3355.128504] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3356.882218] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3359.087093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.204263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.101197] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3362.275193] LustreError: 241058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.278042] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3362.282045] LustreError: 241058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3363.412482] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3365.665419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.732268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.550571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3369.847326] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3370.982615] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3372.008762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3372.009057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3375.669885] Lustre: Permanently reactivating lustre-MDT0001 [ 3375.675300] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3375.677478] Lustre: Skipped 2 previous similar messages [ 3378.789686] Lustre: lustre-MDT0001: Client 40014b3b-ca0e-4385-a91b-e1d3417f3dab (at 192.168.204.44@tcp) reconnecting [ 3380.607176] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3381.194371] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3382.832109] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3382.899772] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3384.193910] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3384.268868] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3385.083994] Lustre: Permanently deactivating lustre-MDT0001 [ 3385.089849] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3385.091703] Lustre: Skipped 2 previous similar messages [ 3391.484954] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3396.065635] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3396.068796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.074518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.624779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.625622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.626732] Lustre: Skipped 1 previous similar message [ 3398.630394] Lustre: Skipped 2 previous similar messages [ 3401.177165] Lustre: server umount lustre-MDT0000 complete [ 3402.370162] LustreError: 239953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931056 with bad export cookie 3900079033149592682 [ 3402.374232] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.375253] LustreError: 239953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3402.493608] Lustre: server umount lustre-MDT0001 complete [ 3409.784657] Lustre: server umount lustre-OST0000 complete [ 3413.429696] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:37:46 (1782931066) [ 3416.750529] Lustre: server umount lustre-OST0001 complete [ 3418.761148] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3421.961862] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3433.244436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.324739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3433.327639] Lustre: Skipped 2 previous similar messages [ 3433.335341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3433.371510] Lustre: lustre-MDT0000: new disk, initializing [ 3433.391831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.398582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3434.766238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3438.930532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.974369] Lustre: 247507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3438.997268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3438.999213] Lustre: Skipped 1 previous similar message [ 3439.040060] Lustre: lustre-MDT0001: new disk, initializing [ 3439.088598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3439.091539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3440.492279] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3444.304976] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3446.199757] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3448.521933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.632517] Lustre: lustre-OST0000: new disk, initializing [ 3448.634214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3448.637131] Lustre: 249474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3448.668444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.670428] Lustre: Skipped 1 previous similar message [ 3450.487565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3450.492970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3450.529781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3450.575555] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3454.262749] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3454.912064] LustreError: 248221:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3459.138590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.209452] Lustre: lustre-OST0001: new disk, initializing [ 3459.211925] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3459.215473] Lustre: 250590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3459.904583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3459.908890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3459.948828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3461.267610] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3465.007064] LustreError: 248221:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3465.016199] LustreError: 247514:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3465.034138] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3465.585391] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3475.103075] LustreError: 247514:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3476.648840] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 3480.547249] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.550285] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.554298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3480.557709] Lustre: Skipped 1 previous similar message [ 3481.852147] Lustre: server umount lustre-OST0001 complete [ 3485.668243] LustreError: 249458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.675214] LustreError: 249458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3490.787172] LustreError: 249459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.793561] LustreError: 249459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3495.906199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.906462] LustreError: 249458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.907866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.907874] Lustre: Skipped 1 previous similar message [ 3495.909476] Lustre: Skipped 2 previous similar messages [ 3495.913857] LustreError: 249458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3501.027646] LustreError: 249459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.032543] LustreError: 249459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3506.146834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.149398] Lustre: Skipped 4 previous similar messages [ 3508.191144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3508.251297] Lustre: server umount lustre-OST0000 complete [ 3510.751881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3510.756434] LustreError: Skipped 1 previous similar message [ 3510.758190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.671084] Lustre: server umount lustre-MDT0000 complete [ 3516.383900] LustreError: 247520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.388560] LustreError: 247520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3516.793073] LustreError: 247500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931170 with bad export cookie 3900079033149595468 [ 3516.796645] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.798195] LustreError: 247500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3531.231104] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3531.321549] Lustre: server umount lustre-MDT0001 complete [ 3534.833157] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3535.829371] Key type lgssc unregistered [ 3535.960290] LNet: 252624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3535.962714] LNetError: 252624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3535.972324] LNet: Removed LNI 192.168.204.144@tcp [ 3536.265585] Key type .llcrypt unregistered [ 3536.266692] Key type ._llcrypt unregistered [ 3549.752265] Key type ._llcrypt registered [ 3549.753627] Key type .llcrypt registered [ 3549.789651] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3550.054205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3550.066874] alg: No test for adler32 (adler32-zlib) [ 3550.924554] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3551.013848] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3552.591105] Key type lgssc registered [ 3552.937597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.421363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.427184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.510141] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3556.516392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.569846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.859634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3560.585050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.619619] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3560.629687] Lustre: 254808:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3560.723209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3562.022156] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3564.335096] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3566.144111] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3568.363125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3568.439652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3568.514845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.484738] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3572.114958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3572.829286] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.153363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3613.154036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.155471] LustreError: Skipped 1 previous similar message [ 3613.155987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3613.159952] Lustre: Skipped 1 previous similar message [ 3618.011312] Lustre: server umount lustre-OST0000 complete [ 3619.411487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3623.438280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.549917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.377481] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3627.646582] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3632.803134] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3632.808477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3632.808643] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3632.814785] Lustre: Skipped 1 previous similar message [ 3663.841262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.842473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.843425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.843429] Lustre: Skipped 1 previous similar message [ 3663.843980] LustreError: Skipped 1 previous similar message [ 3663.850259] Lustre: Skipped 1 previous similar message [ 3666.841464] Lustre: server umount lustre-OST0000 complete [ 3668.447894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3668.450058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.453541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.455105] Lustre: Skipped 1 previous similar message [ 3674.080309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.082103] Lustre: Skipped 2 previous similar messages [ 3679.200166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.202145] Lustre: Skipped 1 previous similar message [ 3682.271153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3682.329116] Lustre: server umount lustre-MDT0000 complete [ 3683.501213] LustreError: 255547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931337 with bad export cookie 11591724000560331902 [ 3683.503893] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.505056] LustreError: 255547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3683.622704] Lustre: server umount lustre-MDT0001 complete [ 3687.254620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3688.294852] Key type lgssc unregistered [ 3688.433385] LNet: 260217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.435690] LNetError: 260217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.441354] LNet: Removed LNI 192.168.204.144@tcp [ 3688.776722] Key type .llcrypt unregistered [ 3688.777831] Key type ._llcrypt unregistered [ 3698.191281] Key type ._llcrypt registered [ 3698.192568] Key type .llcrypt registered [ 3698.232715] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3698.615118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3698.645434] alg: No test for adler32 (adler32-zlib) [ 3699.491864] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3699.577959] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3701.159169] Key type lgssc registered [ 3701.533038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.064952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.072223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.221529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.575091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3709.306344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.436696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3710.731974] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3713.012252] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3714.766788] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3716.960590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.070401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.112111] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3721.565253] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.409374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3740.641858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.642673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.645559] Lustre: Skipped 1 previous similar message [ 3740.645815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.647450] Lustre: Skipped 1 previous similar message [ 3745.552988] Lustre: server umount lustre-OST0000 complete [ 3750.368211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3750.374288] LustreError: Skipped 1 previous similar message [ 3750.377183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.387161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.001328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.006691] Lustre: Skipped 2 previous similar messages [ 3761.119104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3761.120543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.123417] Lustre: Skipped 1 previous similar message [ 3761.184482] Lustre: server umount lustre-MDT0000 complete [ 3762.613051] LustreError: 261591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931416 with bad export cookie 9035141265822024322 [ 3762.618595] LustreError: 261591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3762.620933] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.764471] Lustre: server umount lustre-MDT0001 complete [ 3766.510196] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3767.649398] Key type lgssc unregistered [ 3767.774426] LNet: 265931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3767.776840] LNetError: 265931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3767.785356] LNet: Removed LNI 192.168.204.144@tcp [ 3768.086959] Key type .llcrypt unregistered [ 3768.088111] Key type ._llcrypt unregistered [ 3775.671241] Key type ._llcrypt registered [ 3775.672217] Key type .llcrypt registered [ 3775.709623] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3776.016369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3776.044499] alg: No test for adler32 (adler32-zlib) [ 3776.897633] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3776.983310] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3778.559189] Key type lgssc registered [ 3778.908472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3781.713473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3781.722672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.887479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.296803] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3787.152620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.283935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3788.752588] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3791.133979] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3792.875614] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3795.100614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.224835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.161984] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3799.501487] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3808.561567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3813.856481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.859430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.862909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.865087] Lustre: Skipped 1 previous similar message [ 3818.978595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3818.980515] Lustre: Skipped 1 previous similar message [ 3824.098726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.100922] Lustre: Skipped 1 previous similar message [ 3828.191163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3828.249107] Lustre: server umount lustre-OST0000 complete [ 3833.312261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3833.317155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.325800] Lustre: Skipped 1 previous similar message [ 3833.328851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.338891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.698318] Lustre: server umount lustre-MDT0000 complete [ 3837.431878] LustreError: 267111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931491 with bad export cookie 778347818619724446 [ 3837.438146] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3837.439817] LustreError: 267111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3837.584109] Lustre: server umount lustre-MDT0001 complete [ 3841.275452] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3842.293236] Key type lgssc unregistered [ 3842.412417] LNet: 270825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3842.415476] LNetError: 270825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3842.425370] LNet: Removed LNI 192.168.204.144@tcp [ 3842.746568] Key type .llcrypt unregistered [ 3842.748032] Key type ._llcrypt unregistered [ 3851.614179] Key type ._llcrypt registered [ 3851.615831] Key type .llcrypt registered [ 3851.654937] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3851.975422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3852.027381] alg: No test for adler32 (adler32-zlib) [ 3852.872455] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3852.956788] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3854.535162] Key type lgssc registered [ 3854.897966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3857.420198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3857.425825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.569124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.816536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3863.312238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.473470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3864.898856] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3867.798436] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3869.864855] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3872.494514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.623296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.661440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3877.065924] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3883.878558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3895.778900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3895.784134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.792343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.361106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.363794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.363939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.135857] Lustre: server umount lustre-OST0000 complete [ 3904.481493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.481527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3904.485853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.491887] Lustre: Skipped 2 previous similar messages [ 3908.507784] Lustre: server umount lustre-MDT0000 complete [ 3909.600058] LustreError: 272945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.605135] LustreError: 272945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3909.782365] LustreError: 272952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931563 with bad export cookie 8921221993332565572 [ 3909.784896] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3909.785877] LustreError: 272952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3909.921796] Lustre: server umount lustre-MDT0001 complete [ 3913.410739] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3914.396170] Key type lgssc unregistered [ 3914.519361] LNet: 276539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3914.521670] LNetError: 276539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3914.530311] LNet: Removed LNI 192.168.204.144@tcp [ 3914.831659] Key type .llcrypt unregistered [ 3914.833070] Key type ._llcrypt unregistered [ 3921.636214] Key type ._llcrypt registered [ 3921.637296] Key type .llcrypt registered [ 3921.676091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3922.010127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.048870] alg: No test for adler32 (adler32-zlib) [ 3922.901696] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3922.994174] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3924.575210] Key type lgssc registered [ 3924.933095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3927.577295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3927.584972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.759755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.114386] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3933.108197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.262441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.713884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3937.059360] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3938.819450] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3941.320063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3941.460523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.583505] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3946.244800] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.765377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3963.360336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3963.364066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.369752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.384971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3964.385870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.388137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.825159] Lustre: server umount lustre-OST0000 complete [ 3968.991950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3968.994450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.999526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.001593] Lustre: Skipped 1 previous similar message [ 3974.229455] Lustre: server umount lustre-MDT0000 complete [ 3974.623991] LustreError: 277752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.630059] LustreError: 277752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3975.435365] LustreError: 277717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931629 with bad export cookie 11595671528365893118 [ 3975.438601] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.441326] LustreError: 277717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3975.552741] Lustre: server umount lustre-MDT0001 complete [ 3979.122303] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3980.129272] Key type lgssc unregistered [ 3980.257332] LNet: 281468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3980.260589] LNetError: 281468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.268307] LNet: Removed LNI 192.168.204.144@tcp [ 3980.564601] Key type .llcrypt unregistered [ 3980.566589] Key type ._llcrypt unregistered [ 3984.301348] Key type ._llcrypt registered [ 3984.302915] Key type .llcrypt registered [ 3984.344148] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3989.707245] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3995.581650] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3995.915684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3995.977934] alg: No test for adler32 (adler32-zlib) [ 3996.820573] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 3996.902858] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3998.487128] Key type lgssc registered [ 3998.897695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4010.538627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4010.544938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.631365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4011.644086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4011.684883] Lustre: lustre-MDT0000: new disk, initializing [ 4011.705349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.712944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4013.039340] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4017.119589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.160847] Lustre: 284991:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4017.183268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4017.184951] Lustre: Skipped 1 previous similar message [ 4017.220116] Lustre: lustre-MDT0001: new disk, initializing [ 4017.251055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.262766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4017.265858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4018.626924] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4022.282333] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4024.005205] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4026.144215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.250300] Lustre: lustre-OST0000: new disk, initializing [ 4026.252756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4026.256508] Lustre: 286958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4026.292416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.348369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4026.352716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4026.363199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4028.258262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4031.999675] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4033.442765] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4033.512783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4034.817738] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4034.898325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4036.580764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4036.580792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.583518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.588840] Lustre: Skipped 1 previous similar message [ 4037.583499] Lustre: server umount lustre-OST0000 complete [ 4041.696842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.697715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.700813] Lustre: Skipped 1 previous similar message [ 4041.704569] Lustre: Skipped 2 previous similar messages [ 4045.023296] Lustre: server umount lustre-MDT0000 complete [ 4046.200296] LustreError: 284985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931699 with bad export cookie 13749830755232484720 [ 4046.203396] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.205753] LustreError: 284985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4046.317069] Lustre: server umount lustre-MDT0001 complete [ 4048.955601] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:48:22 (1782931702) [ 4051.256494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.463919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.875348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4055.680628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.798566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.126094] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4059.437758] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4061.220027] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4063.557246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4065.574371] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4067.828721] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4081.119793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4081.122189] LustreError: Skipped 1 previous similar message [ 4081.123539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.127942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.152715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.189386] Lustre: server umount lustre-OST0000 complete [ 4091.872223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.873752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.874787] LustreError: Skipped 1 previous similar message [ 4091.878604] Lustre: Skipped 1 previous similar message [ 4091.881913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.884448] Lustre: Skipped 2 previous similar messages [ 4094.753585] Lustre: server umount lustre-MDT0000 complete [ 4096.059440] LustreError: 289346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931749 with bad export cookie 13749830755232485588 [ 4096.062157] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4096.065212] LustreError: 289346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4096.195576] Lustre: server umount lustre-MDT0001 complete [ 4100.108175] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4101.138949] Key type lgssc unregistered [ 4101.283377] LNet: 292928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4101.286613] LNetError: 292928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4101.293436] LNet: Removed LNI 192.168.204.144@tcp [ 4101.601551] Key type .llcrypt unregistered [ 4101.602919] Key type ._llcrypt unregistered [ 4110.335692] Key type ._llcrypt registered [ 4110.337491] Key type .llcrypt registered [ 4110.377700] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4116.661643] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4117.065299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4117.129476] alg: No test for adler32 (adler32-zlib) [ 4117.995946] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4118.095512] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4119.679201] Key type lgssc registered [ 4120.100164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4141.131894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4141.143051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.296213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4142.314419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4142.367416] Lustre: lustre-MDT0000: new disk, initializing [ 4142.411180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.425761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4144.509963] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4150.110873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.167670] Lustre: 296726:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4150.217422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4150.221265] Lustre: Skipped 1 previous similar message [ 4150.269150] Lustre: lustre-MDT0001: new disk, initializing [ 4150.312370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4150.330739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4150.335646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4152.129320] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4156.777820] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4158.992797] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4161.814355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.935606] Lustre: lustre-OST2710: new disk, initializing [ 4161.938540] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4161.943123] Lustre: 298694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4161.980527] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4164.193160] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4168.569875] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4168.706491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4168.712257] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4168.763474] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4171.488462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.569921] Lustre: lustre-OST03e8: new disk, initializing [ 4171.572542] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4171.577662] Lustre: 299756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4171.623368] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4174.096489] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4177.179395] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4178.976490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4178.983494] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4179.038720] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4180.918232] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4181.027520] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4182.761207] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4182.843768] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4184.566272] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4184.651655] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4186.004522] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4186.069199] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4189.159126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.160510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.162679] Lustre: Skipped 3 previous similar messages [ 4189.164549] Lustre: Skipped 2 previous similar messages [ 4193.176038] Lustre: server umount lustre-MDT0000 complete [ 4194.273382] LustreError: 296736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.279661] LustreError: 296736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4194.458130] LustreError: 296718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931848 with bad export cookie 9818526006024377450 [ 4194.460899] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.461944] LustreError: 296718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4194.595627] Lustre: server umount lustre-MDT0001 complete [ 4205.572458] Lustre: server umount lustre-OST2710 complete [ 4217.052881] Lustre: server umount lustre-OST03e8 complete [ 4223.138579] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4226.396946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4238.336572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.436738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4238.451588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4238.491860] Lustre: lustre-MDT0000: new disk, initializing [ 4238.524796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.536219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.203905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4244.778991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.824298] Lustre: 304997:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.929836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4246.462855] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4250.485623] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4252.616384] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4255.221664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.324915] Lustre: lustre-OST0000: new disk, initializing [ 4255.326956] Lustre: Skipped 1 previous similar message [ 4255.329256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4255.332150] Lustre: Skipped 2 previous similar messages [ 4255.334931] Lustre: 306964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4255.367161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.370103] Lustre: Skipped 1 previous similar message [ 4256.958938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4256.962568] Lustre: Skipped 1 previous similar message [ 4256.965054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4257.005714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4257.500854] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4261.348982] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4262.943786] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4263.018928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4264.471322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4264.545449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4267.491948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4267.492924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.496320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4267.502039] Lustre: Skipped 1 previous similar message [ 4271.389540] Lustre: server umount lustre-OST0000 complete [ 4275.680086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4275.683856] LustreError: Skipped 1 previous similar message [ 4275.685489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.691389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.693867] Lustre: Skipped 1 previous similar message [ 4277.728126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.728755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.944475] Lustre: server umount lustre-MDT0000 complete [ 4280.241229] LustreError: 305789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931933 with bad export cookie 9818526006024379123 [ 4280.243416] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.246140] LustreError: 305789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4280.366903] Lustre: server umount lustre-MDT0001 complete [ 4283.467295] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:52:16 (1782931936) [ 4284.060897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4284.716770] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:52:18 (1782931938) [ 4285.877490] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4295.312664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4295.397290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4295.403616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4295.467993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.819276] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4299.785529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.822355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4299.833887] Lustre: 310816:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4301.270900] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4317.345853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4317.405940] Lustre: Denying initial registration attempt from nid 192.168.204.144@tcp, specified as failover [ 4317.408846] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4317.411345] LustreError: 312269:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4317.413528] LustreError: 312269:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4317.415329] LustreError: 312269:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4317.437279] Lustre: server umount lustre-OST0000 complete [ 4317.439084] LustreError: 312269:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4320.735688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4320.737927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.741517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.742923] Lustre: Skipped 1 previous similar message [ 4325.529467] Lustre: server umount lustre-MDT0000 complete [ 4326.368093] LustreError: 310826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.374457] LustreError: 310826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4326.781500] LustreError: 310807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782931980 with bad export cookie 9818526006024379991 [ 4326.784765] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.786527] LustreError: 310807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.904693] Lustre: server umount lustre-MDT0001 complete [ 4330.437896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4331.440535] Key type lgssc unregistered [ 4331.571357] LNet: 313388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4331.574325] LNetError: 313388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4331.583272] LNet: Removed LNI 192.168.204.144@tcp [ 4331.897583] Key type .llcrypt unregistered [ 4331.898605] Key type ._llcrypt unregistered [ 4335.562097] Key type ._llcrypt registered [ 4335.563650] Key type .llcrypt registered [ 4335.607228] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4336.050688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4336.064307] alg: No test for adler32 (adler32-zlib) [ 4336.922618] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4337.012875] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4338.599135] Key type lgssc registered [ 4338.971559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4362.549473] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4365.478952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.484662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.573435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4366.580115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4366.633623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.954456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4370.637568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.668564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4370.678207] Lustre: 316126:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4370.766233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4372.013838] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4374.164579] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.837319] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4390.946761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4391.027839] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4391.129334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.187179] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4395.400587] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4396.171567] Lustre: server umount lustre-OST0000 complete [ 4399.603248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.604299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.607594] Lustre: Skipped 1 previous similar message [ 4403.666536] Lustre: server umount lustre-MDT0000 complete [ 4404.866297] LustreError: 316118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932058 with bad export cookie 8705675853562974497 [ 4404.869657] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.869863] LustreError: 316118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4404.984312] Lustre: server umount lustre-MDT0001 complete [ 4408.419201] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4409.378427] Key type lgssc unregistered [ 4409.501441] LNet: 319687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4409.505125] LNetError: 319687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4409.516280] LNet: Removed LNI 192.168.204.144@tcp [ 4409.824127] Key type .llcrypt unregistered [ 4409.825250] Key type ._llcrypt unregistered [ 4418.904545] Key type ._llcrypt registered [ 4418.905761] Key type .llcrypt registered [ 4418.944100] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4419.374165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4419.380268] alg: No test for adler32 (adler32-zlib) [ 4420.226545] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4420.312973] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4421.895169] Key type lgssc registered [ 4422.236356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4424.654121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4424.661824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.795966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.054917] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4429.631219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.754307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4430.989139] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4433.147408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4434.933267] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4439.650683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4439.779804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.616985] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4444.069437] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4459.688051] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 4460.511956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4460.514752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.518936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.538010] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.540477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.543738] Lustre: Skipped 1 previous similar message [ 4461.551556] Lustre: Skipped 1 previous similar message [ 4463.025741] Lustre: server umount lustre-MDT0000 complete [ 4464.443522] LustreError: 321062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932118 with bad export cookie 5494160268386528636 [ 4464.448515] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4464.449598] LustreError: 321062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4464.798943] LustreError: 321075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.810547] LustreError: 321075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4464.811176] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 4466.656104] LustreError: 321076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.656286] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.663835] LustreError: 321075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.664162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4469.672030] Lustre: Skipped 1 previous similar message [ 4470.636802] Lustre: server umount lustre-MDT0001 complete [ 4472.763184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4475.471346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.207758] LustreError: 320392:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000098f10bb4 x1869539781403008/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4490.326577] LustreError: 322907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.331421] LustreError: 322907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4490.351488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.370365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4491.630806] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4494.313682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.433569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.460950] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4495.720080] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4497.943199] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4499.556469] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4500.505821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4502.186807] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4502.696824] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4504.544278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4504.549061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.557404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4504.560465] Lustre: Skipped 1 previous similar message [ 4518.367081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4518.417344] Lustre: server umount lustre-OST0000 complete [ 4519.903933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4519.906040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.909057] Lustre: Skipped 1 previous similar message [ 4525.536227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.538013] Lustre: Skipped 10 previous similar messages [ 4525.717653] Lustre: server umount lustre-MDT0000 complete [ 4526.911052] LustreError: 321061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932180 with bad export cookie 5494160268386535790 [ 4526.914396] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.915990] LustreError: 321061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4527.024954] Lustre: server umount lustre-MDT0001 complete [ 4530.396560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4531.324567] Key type lgssc unregistered [ 4531.456302] LNet: 327361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4531.458840] LNetError: 327361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4531.466334] LNet: Removed LNI 192.168.204.144@tcp [ 4531.765893] Key type .llcrypt unregistered [ 4531.767021] Key type ._llcrypt unregistered [ 4540.293873] Key type ._llcrypt registered [ 4540.294997] Key type .llcrypt registered [ 4540.328612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4540.648132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4540.700634] alg: No test for adler32 (adler32-zlib) [ 4541.555619] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4541.638892] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4543.215130] Key type lgssc registered [ 4543.556301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4546.012485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4546.018358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.165835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.417275] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4551.045053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.170070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4552.389262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4554.613729] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4556.298444] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4561.312772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.318833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.320841] Lustre: Skipped 1 previous similar message [ 4563.347829] Lustre: server umount lustre-MDT0000 complete [ 4564.496193] LustreError: 328735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932218 with bad export cookie 1595654011770768581 [ 4564.499990] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.500645] LustreError: 328735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.609770] Lustre: server umount lustre-MDT0001 complete [ 4569.084269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.180294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4569.191430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.261684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.498879] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4573.141820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.176034] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4573.187785] Lustre: 331340:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4573.282553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.552130] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4576.801336] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4578.459914] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4582.882409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.952875] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4582.956108] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4582.959235] LustreError: 333224:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4582.961637] LustreError: 333224:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4582.963836] LustreError: 333224:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4582.986134] Lustre: server umount lustre-OST0000 complete [ 4582.988260] LustreError: 333224:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4587.749932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.844169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4587.905464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.903398] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4592.175259] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4594.184843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4594.565323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.594424] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4594.623467] Lustre: lustre-OST0001: new disk, initializing [ 4594.625329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4594.628061] Lustre: 334760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4596.457469] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4598.708568] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4599.654725] Lustre: server umount lustre-OST0001 complete [ 4600.316336] LustreError: 334679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.320683] LustreError: 334679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4604.384123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4604.384284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.387674] LustreError: Skipped 1 previous similar message [ 4604.391503] Lustre: Skipped 1 previous similar message [ 4604.392077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4604.396680] Lustre: Skipped 1 previous similar message [ 4605.409960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4605.411351] LustreError: 333691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.411871] Lustre: Skipped 1 previous similar message [ 4605.415407] LustreError: 333691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4606.984298] Lustre: server umount lustre-OST0000 complete [ 4608.991821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4608.995634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.001344] Lustre: Skipped 1 previous similar message [ 4609.003656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.298884] Lustre: server umount lustre-MDT0000 complete [ 4615.436956] LustreError: 332080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932269 with bad export cookie 1595654011770769120 [ 4615.439426] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.440057] LustreError: 332080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4615.553337] Lustre: server umount lustre-MDT0001 complete [ 4618.840102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4619.779277] Key type lgssc unregistered [ 4619.892310] LNet: 336712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4619.894399] LNetError: 336712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4619.903281] LNet: Removed LNI 192.168.204.144@tcp [ 4620.201725] Key type .llcrypt unregistered [ 4620.203566] Key type ._llcrypt unregistered [ 4649.831732] Key type ._llcrypt registered [ 4649.832720] Key type .llcrypt registered [ 4649.865671] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4655.505916] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4656.031381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.043853] alg: No test for adler32 (adler32-zlib) [ 4656.891730] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4656.978295] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4658.559199] Key type lgssc registered [ 4658.940302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4669.631318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4669.639080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.738757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.749574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.790397] Lustre: lustre-MDT0000: new disk, initializing [ 4670.818143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.825509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4672.280940] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4676.599466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.648450] Lustre: 341612:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.678969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4676.681692] Lustre: Skipped 1 previous similar message [ 4676.723367] Lustre: lustre-MDT0001: new disk, initializing [ 4676.759942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4676.775220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4676.779693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4678.188649] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4682.093503] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4684.123296] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.788248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.924905] Lustre: lustre-OST0000: new disk, initializing [ 4686.926868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4686.929883] Lustre: 343582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4686.964762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.009531] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4689.855218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4689.857967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4689.899035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4692.796219] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4694.405876] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4694.505257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4696.036665] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4696.132310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4700.130672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.131764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.132686] LustreError: Skipped 1 previous similar message [ 4700.136377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4700.138519] Lustre: Skipped 1 previous similar message [ 4705.251748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4705.253425] Lustre: Skipped 1 previous similar message [ 4710.370647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.372392] Lustre: Skipped 1 previous similar message [ 4711.391117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4711.440706] Lustre: server umount lustre-OST0000 complete [ 4712.928180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4712.931475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.936908] Lustre: Skipped 1 previous similar message [ 4712.939461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4715.489178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.742635] Lustre: server umount lustre-MDT0000 complete [ 4719.887166] LustreError: 341604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932373 with bad export cookie 5495207428364466950 [ 4719.890451] LustreError: 341604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4719.891046] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.004109] Lustre: server umount lustre-MDT0001 complete [ 4722.539331] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:59:35 (1782932375) [ 4724.743137] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:59:38 (1782932378) [ 4727.395057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.562947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.775152] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4731.415841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.756750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4734.942733] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4736.595299] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4738.711581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4738.830893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.833399] Lustre: Skipped 1 previous similar message [ 4740.708415] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4742.932320] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4756.982291] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4757.641572] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4758.244363] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4758.827711] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4759.660847] Lustre: Failing over lustre-MDT0000 [ 4759.875048] Lustre: server umount lustre-MDT0000 complete [ 4762.592906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4762.597450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.606150] LustreError: 346350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.613930] LustreError: 346350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4763.617364] LustreError: 346331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.629019] LustreError: 346331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4768.736677] LustreError: 346332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.745881] LustreError: 346332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4773.373304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.440942] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4773.569630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.605496] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4774.045789] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4775.193427] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4778.983896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4778.995575] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4779.017187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4780.761482] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4781.340528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4783.237969] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4783.873096] Lustre: Failing over lustre-MDT0000 [ 4783.989537] Lustre: server umount lustre-MDT0000 complete [ 4784.097668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.098032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4784.099313] LustreError: 346328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.099325] LustreError: 346328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4784.104250] Lustre: Skipped 4 previous similar messages [ 4789.216061] LustreError: 346332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.223066] LustreError: 346332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4796.877054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.923408] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4797.023650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.049462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4798.403218] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4799.645459] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4802.535692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4802.537788] Lustre: Skipped 2 previous similar messages [ 4802.550824] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4802.571262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4804.282373] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4804.794034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4807.650319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.651245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4807.651314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4807.657022] Lustre: Skipped 2 previous similar messages [ 4807.659882] Lustre: Skipped 3 previous similar messages [ 4813.137760] Lustre: server umount lustre-MDT0000 complete [ 4814.306118] LustreError: 346312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932467 with bad export cookie 5495207428364469645 [ 4814.308994] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4814.311609] LustreError: 346312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4814.417819] Lustre: server umount lustre-MDT0001 complete [ 4825.946716] Lustre: server umount lustre-OST0000 complete [ 4830.788176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.940042] LustreError: 352690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.949223] LustreError: 352690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4830.962352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.227792] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4834.952349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.356071] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4838.631910] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4840.313229] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4842.444687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.348153] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4846.533394] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4856.871046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4859.872064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4859.875023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.880977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4859.882618] Lustre: Skipped 4 previous similar messages [ 4864.400663] Lustre: server umount lustre-OST0000 complete [ 4871.784567] Lustre: server umount lustre-MDT0000 complete [ 4872.160852] LustreError: 352709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.165273] LustreError: 352709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4872.937995] LustreError: 352671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932526 with bad export cookie 5495207428364470366 [ 4872.940752] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4872.941239] LustreError: 352671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4873.051230] Lustre: server umount lustre-MDT0001 complete [ 4877.629488] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4878.743123] Key type lgssc unregistered [ 4878.879341] LNet: 356239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4878.881763] LNetError: 356239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4878.891291] LNet: Removed LNI 192.168.204.144@tcp [ 4879.199455] Key type .llcrypt unregistered [ 4879.200783] Key type ._llcrypt unregistered [ 4890.960283] Key type ._llcrypt registered [ 4890.961644] Key type .llcrypt registered [ 4891.016365] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4891.437510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4891.502290] alg: No test for adler32 (adler32-zlib) [ 4892.421411] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 4892.520265] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4894.103155] Key type lgssc registered [ 4894.545976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4897.326765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4897.337599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.497392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.950726] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4902.823421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.959566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4904.428640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4907.019585] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4909.132809] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.172632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4912.348363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.870845] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4917.959947] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4923.628167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4933.599890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4933.602842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.608229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4934.112537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4934.114707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.525821] Lustre: server umount lustre-MDT0000 complete [ 4939.233519] LustreError: 357732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.238319] LustreError: 357732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4939.729058] LustreError: 357713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932593 with bad export cookie 9425577946316838388 [ 4939.732321] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.735731] LustreError: 357713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4939.842348] Lustre: server umount lustre-MDT0001 complete [ 4951.327189] Lustre: server umount lustre-OST0000 complete [ 4956.090418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.268887] LustreError: 361463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.292352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.649561] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4960.775566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.922903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4962.498114] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4965.401223] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4967.921105] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4971.075070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4971.172734] LustreError: 363280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4971.241541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.677266] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4977.048799] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4980.716861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4990.948146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.950743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.959537] Lustre: Skipped 3 previous similar messages [ 4990.965068] Lustre: Skipped 3 previous similar messages [ 4995.481215] Lustre: server umount lustre-MDT0000 complete [ 4996.066648] LustreError: 363952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.076065] LustreError: 363952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4997.106349] LustreError: 361445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932650 with bad export cookie 9425577946316839487 [ 4997.113389] LustreError: 361445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4997.113574] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.256424] Lustre: server umount lustre-MDT0001 complete [ 5009.144230] Lustre: server umount lustre-OST0000 complete [ 5015.971215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.212368] LustreError: 365371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.247151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5018.052269] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5021.471736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5023.333015] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5025.808986] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5027.527231] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5029.792238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.940728] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5034.416170] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5041.190116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5046.241266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.241561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.245800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.249013] Lustre: Skipped 1 previous similar message [ 5048.277717] Lustre: server umount lustre-OST0000 complete [ 5051.361279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.362112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.365993] Lustre: Skipped 2 previous similar messages [ 5051.606044] Lustre: server umount lustre-MDT0000 complete [ 5052.879277] LustreError: 366105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932706 with bad export cookie 9425577946316840495 [ 5052.883850] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.884338] LustreError: 366105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5053.009801] Lustre: server umount lustre-MDT0001 complete [ 5056.649281] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5057.622456] Key type lgssc unregistered [ 5057.752369] LNet: 368920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5057.755992] LNetError: 368920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5057.764284] LNet: Removed LNI 192.168.204.144@tcp [ 5058.077403] Key type .llcrypt unregistered [ 5058.078514] Key type ._llcrypt unregistered [ 5067.488578] Key type ._llcrypt registered [ 5067.489797] Key type .llcrypt registered [ 5067.527305] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5067.954675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5067.962803] alg: No test for adler32 (adler32-zlib) [ 5068.809690] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5068.903965] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5070.479136] Key type lgssc registered [ 5070.831992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5073.310354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5073.316584] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.320285] LustreError: 370337:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5073.328879] LustreError: 370337:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5073.332124] LustreError: 370337:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5073.335649] LustreError: 370337:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5073.339556] LustreError: 370337:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5075.658763] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.661812] LustreError: 370734:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5075.666827] LustreError: 370734:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5075.668945] LustreError: 370734:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5075.670963] LustreError: 370734:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5075.673404] LustreError: 370734:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5081.536564] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5082.484704] Key type lgssc unregistered [ 5082.597354] LNet: 371734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5082.599805] LNetError: 371734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5082.608246] LNet: Removed LNI 192.168.204.144@tcp [ 5082.899293] Key type .llcrypt unregistered [ 5082.900334] Key type ._llcrypt unregistered [ 5088.416678] Key type ._llcrypt registered [ 5088.417683] Key type .llcrypt registered [ 5088.453058] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5094.298231] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5094.726710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5094.744236] alg: No test for adler32 (adler32-zlib) [ 5095.592104] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5095.681869] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5097.263135] Key type lgssc registered [ 5097.612571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5108.792686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5108.802565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.918102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5109.928033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5109.973828] Lustre: lustre-MDT0000: new disk, initializing [ 5109.996190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.002094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5111.697653] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5116.708918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.782466] Lustre: 374864:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5116.816575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5116.820098] Lustre: Skipped 1 previous similar message [ 5116.865362] Lustre: lustre-MDT0001: new disk, initializing [ 5116.907916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5116.932046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5116.938544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5118.306385] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5122.329237] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5124.239473] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5126.602448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.699410] Lustre: lustre-OST0000: new disk, initializing [ 5126.701046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5126.703441] Lustre: 376831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5126.731649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.623029] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5132.448669] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.292105] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5134.852918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5134.858503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5134.904382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5135.454592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5137.151518] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5137.239233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5139.939725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5139.940643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.943622] LustreError: Skipped 1 previous similar message [ 5139.949870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.953767] Lustre: Skipped 1 previous similar message [ 5144.103992] Lustre: server umount lustre-OST0000 complete [ 5147.616613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5147.623092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.631844] Lustre: Skipped 1 previous similar message [ 5147.635515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.177502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.178449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5151.936926] Lustre: server umount lustre-MDT0000 complete [ 5153.200414] LustreError: 374859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932806 with bad export cookie 14228148116593586255 [ 5153.203677] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.204828] LustreError: 374859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.330501] Lustre: server umount lustre-MDT0001 complete [ 5156.938385] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5157.802731] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:06:50 (1782932810) [ 5160.950709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.180272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.829463] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5165.976022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.105483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5167.658054] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5170.208936] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5172.276273] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5174.697295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.810809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.913807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5179.329652] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5181.593152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5181.680187] Lustre: lustre-OST0001: new disk, initializing [ 5181.682183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5181.685207] Lustre: 382129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5183.899759] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5186.570433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5186.573675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5186.623509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5187.551840] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5191.649499] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.649835] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.652860] LustreError: Skipped 1 previous similar message [ 5191.657017] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5191.659636] Lustre: Skipped 2 previous similar messages [ 5195.515126] Lustre: server umount lustre-OST0001 complete [ 5197.794316] LustreError: 382848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.799257] LustreError: 382848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5202.916293] LustreError: 381291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.920655] LustreError: 381291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5208.035240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5208.037819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.037991] LustreError: 381110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.054428] Lustre: Skipped 1 previous similar message [ 5208.057960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5208.060947] Lustre: Skipped 4 previous similar messages [ 5213.155293] LustreError: 381110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.161608] LustreError: 381110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5213.647863] Lustre: server umount lustre-OST0000 complete [ 5217.247924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5217.250581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.253963] Lustre: Skipped 1 previous similar message [ 5217.255603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.257832] Lustre: Skipped 2 previous similar messages [ 5221.081244] Lustre: server umount lustre-MDT0000 complete [ 5222.242507] LustreError: 381133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932875 with bad export cookie 14228148116593587123 [ 5222.245252] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.245915] LustreError: 381133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5228.404915] Lustre: server umount lustre-MDT0001 complete [ 5231.956661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5232.930561] Key type lgssc unregistered [ 5233.069341] LNet: 384111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5233.071731] LNetError: 384111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5233.079276] LNet: Removed LNI 192.168.204.144@tcp [ 5233.387403] Key type .llcrypt unregistered [ 5233.388466] Key type ._llcrypt unregistered [ 5250.873426] Key type ._llcrypt registered [ 5250.874645] Key type .llcrypt registered [ 5250.908612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5251.214514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5251.272465] alg: No test for adler32 (adler32-zlib) [ 5252.129555] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5252.219839] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5253.799112] Key type lgssc registered [ 5254.158288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5256.705644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5256.713208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.804597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5257.813144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5257.875606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.287443] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5262.003318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.033165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5262.043770] Lustre: 386883:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5262.145316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5263.475477] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5265.722352] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5267.403103] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5272.033501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.033976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5272.036586] Lustre: Skipped 1 previous similar message [ 5274.197497] Lustre: server umount lustre-MDT0000 complete [ 5275.362453] LustreError: 387624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932929 with bad export cookie 12462581914836951077 [ 5275.365912] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.366072] LustreError: 387624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5275.486554] Lustre: server umount lustre-MDT0001 complete [ 5278.388783] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:08:51 (1782932931) [ 5280.591846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.795736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5282.088614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5284.674969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.791077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5285.997752] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5288.204134] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5289.868059] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5292.023135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.101340] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5292.179410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.068852] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5296.360210] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5304.067401] Lustre: Permanently deactivating lustre-OST0000 [ 5304.071948] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5312.493183] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5313.506784] LustreError: 392524:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5313.509947] LustreError: 392524:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5313.512021] LustreError: 392524:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5314.064314] LustreError: 392573:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5314.066909] LustreError: 392573:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5320.727094] Lustre: server umount lustre-OST0000 complete [ 5321.572455] LustreError: 392823:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5321.574952] LustreError: 392823:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5323.232629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.233114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.235897] Lustre: Skipped 1 previous similar message [ 5323.237370] Lustre: Skipped 1 previous similar message [ 5328.352078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5328.354055] Lustre: Skipped 2 previous similar messages [ 5331.447446] Lustre: server umount lustre-MDT0000 complete [ 5332.543959] LustreError: 389416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782932986 with bad export cookie 12462581914836951644 [ 5332.546828] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.547192] LustreError: 389416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5342.729462] Lustre: server umount lustre-MDT0001 complete [ 5345.316583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.514742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5347.321488] LustreError: 393980:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5349.179807] Lustre: 394318:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [ 5349.804662] Lustre: server umount MGS complete [ 5353.204334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.346878] LustreError: 395042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.350886] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5353.356020] LustreError: 395042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5353.357762] Lustre: Skipped 1 previous similar message [ 5353.374468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5354.580563] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5357.128609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.220673] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5358.430485] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5360.526686] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5362.086186] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5362.642972] Lustre: Permanently reactivating lustre-OST0000 [ 5362.647265] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5362.649324] Lustre: Skipped 2 previous similar messages [ 5386.719094] LustreError: 396774:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5388.914886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5389.017063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.018913] Lustre: Skipped 1 previous similar message [ 5390.720792] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5392.837289] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5406.687825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5406.690889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.696247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5407.715106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5410.508377] Lustre: server umount lustre-OST0000 complete [ 5412.832675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.833451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.835887] Lustre: Skipped 2 previous similar messages [ 5412.838313] Lustre: Skipped 2 previous similar messages [ 5417.756768] Lustre: server umount lustre-MDT0000 complete [ 5417.951859] LustreError: 395042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.956799] LustreError: 395042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5418.910059] LustreError: 396616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933072 with bad export cookie 12462581914836952680 [ 5418.912732] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.915095] LustreError: 396616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5419.028667] Lustre: server umount lustre-MDT0001 complete [ 5422.185708] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5423.082315] Key type lgssc unregistered [ 5423.194348] LNet: 399663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5423.196421] LNetError: 399663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5423.208307] LNet: Removed LNI 192.168.204.144@tcp [ 5423.511239] Key type .llcrypt unregistered [ 5423.513385] Key type ._llcrypt unregistered [ 5428.856078] Key type ._llcrypt registered [ 5428.857053] Key type .llcrypt registered [ 5428.888881] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5434.601793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5434.906626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5434.969508] alg: No test for adler32 (adler32-zlib) [ 5435.821644] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5435.910115] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5437.503097] Key type lgssc registered [ 5437.918093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5449.086803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5449.114558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.198710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5450.213560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5450.248775] Lustre: lustre-MDT0000: new disk, initializing [ 5450.271373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5450.277179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5451.612154] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5455.695091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.738205] Lustre: 402790:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5455.761778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5455.763301] Lustre: Skipped 1 previous similar message [ 5455.798560] Lustre: lustre-MDT0001: new disk, initializing [ 5455.832849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5455.844354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5455.848426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5457.182641] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5460.809122] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5462.540650] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5464.806784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.942564] Lustre: lustre-OST0000: new disk, initializing [ 5464.944173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5464.946500] Lustre: 404758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5464.977557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.850726] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5470.525658] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5471.943313] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5472.258245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5472.261471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5472.311953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5473.061694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5474.459287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5474.543289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5477.346242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.346490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.348283] LustreError: Skipped 1 previous similar message [ 5477.352604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.354581] Lustre: Skipped 1 previous similar message [ 5481.294909] Lustre: server umount lustre-OST0000 complete [ 5482.464265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.464996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5482.467592] Lustre: Skipped 2 previous similar messages [ 5487.584154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.586404] Lustre: Skipped 2 previous similar messages [ 5488.530909] Lustre: server umount lustre-MDT0000 complete [ 5489.625071] LustreError: 402783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933143 with bad export cookie 17822446997507648546 [ 5489.627928] LustreError: 402783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5489.628487] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.733027] Lustre: server umount lustre-MDT0001 complete [ 5492.124075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5492.623438] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:12:26 (1782933146) [ 5494.857324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.035653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.493190] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5499.709602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.864855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.422807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5503.931880] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.717533] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5508.048701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.193840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.389195] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5512.875903] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5513.526534] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5524.190883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5525.983808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.986899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.991893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5530.081712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.081799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5530.083252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.599506] Lustre: server umount lustre-OST0000 complete [ 5535.201657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.205050] Lustre: Skipped 1 previous similar message [ 5539.110934] Lustre: server umount lustre-MDT0000 complete [ 5540.321573] LustreError: 407219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.326699] LustreError: 407219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5540.363950] LustreError: 407200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933194 with bad export cookie 17822446997507649414 [ 5540.369169] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.515461] Lustre: server umount lustre-MDT0001 complete [ 5543.584907] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5544.498317] Key type lgssc unregistered [ 5544.628405] LNet: 410961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5544.630626] LNetError: 410961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5544.640387] LNet: Removed LNI 192.168.204.144@tcp [ 5544.930339] Key type .llcrypt unregistered [ 5544.931464] Key type ._llcrypt unregistered [ 5552.283843] Key type ._llcrypt registered [ 5552.284988] Key type .llcrypt registered [ 5552.323617] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5552.422186] Key type .llcrypt unregistered [ 5552.423089] Key type ._llcrypt unregistered [ 5559.206544] Key type ._llcrypt registered [ 5559.208507] Key type .llcrypt registered [ 5559.244957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5559.577242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5559.615089] alg: No test for adler32 (adler32-zlib) [ 5560.479432] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5560.589919] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5562.175140] Key type lgssc registered [ 5562.568211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5565.139262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5565.147877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.317472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.686332] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5570.457989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.574884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.608111] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5574.956022] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.649240] LustreError: 413147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.653060] LustreError: 413147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5576.674913] LustreError: 413146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.690438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5577.264385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5577.398922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5578.752760] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5586.911921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.914055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.917764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.937524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.937673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.939781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.758409] Lustre: server umount lustre-OST0000 complete [ 5593.056750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.057078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5593.060550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.063377] Lustre: Skipped 2 previous similar messages [ 5598.176051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.177758] Lustre: Skipped 1 previous similar message [ 5599.085653] Lustre: server umount lustre-MDT0000 complete [ 5600.938153] LustreError: 413128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933254 with bad export cookie 18172201334673898686 [ 5600.945298] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.945906] LustreError: 413128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5601.111814] Lustre: server umount lustre-MDT0001 complete [ 5605.342857] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5606.554537] Key type lgssc unregistered [ 5606.676613] LNet: 416293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5606.679762] LNetError: 416293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5606.687304] LNet: Removed LNI 192.168.204.144@tcp [ 5607.062559] Key type .llcrypt unregistered [ 5607.063637] Key type ._llcrypt unregistered [ 5618.443229] Key type ._llcrypt registered [ 5618.444942] Key type .llcrypt registered [ 5618.483616] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5618.775625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5618.783556] alg: No test for adler32 (adler32-zlib) [ 5619.625747] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5619.706897] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5621.287208] Key type lgssc registered [ 5621.890172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5625.517521] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5625.528081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.991299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5646.111240] LustreError: 417675:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 5672.735268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.242536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5678.459192] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5681.464084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.658718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.692449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5683.233140] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5686.756704] LustreError: 417701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.767756] LustreError: 417701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5686.994956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.168932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5689.107869] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5691.520642] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5693.207688] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5696.992530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5696.997609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.006337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.504555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.509576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.517344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5698.532075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5698.536021] Lustre: Skipped 1 previous similar message [ 5701.482186] Lustre: server umount lustre-OST0000 complete [ 5703.649496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.650907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.658618] Lustre: Skipped 1 previous similar message [ 5708.769115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.774421] Lustre: Skipped 2 previous similar messages [ 5709.427376] Lustre: server umount lustre-MDT0000 complete [ 5710.969493] LustreError: 417683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933364 with bad export cookie 6265794431019469511 [ 5710.972805] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5710.974233] LustreError: 417683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5711.106687] Lustre: server umount lustre-MDT0001 complete [ 5715.232098] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5716.426164] Key type lgssc unregistered [ 5716.554578] LNet: 421242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5716.560138] LNetError: 421242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5716.570598] LNet: Removed LNI 192.168.204.144@tcp [ 5716.908425] Key type .llcrypt unregistered [ 5716.909612] Key type ._llcrypt unregistered [ 5726.959257] Key type ._llcrypt registered [ 5726.960389] Key type .llcrypt registered [ 5727.007467] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5727.315458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5727.325332] alg: No test for adler32 (adler32-zlib) [ 5728.167283] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5728.255733] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5729.839251] Key type lgssc registered [ 5730.267245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.716134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.725640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.895400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.584846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5740.568952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.778831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5742.668498] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5746.078318] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5748.412108] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.758542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.880696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.868285] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5756.612558] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5760.101745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5765.602803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.605158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.610183] Lustre: Skipped 2 previous similar messages [ 5765.615123] Lustre: Skipped 1 previous similar message [ 5766.625791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.631688] Lustre: Skipped 2 previous similar messages [ 5768.364479] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5770.292720] Lustre: server umount lustre-MDT0000 complete [ 5771.342660] LustreError: 422619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1782933425 with bad export cookie 1374335072483022620 [ 5771.352075] LustreError: 422619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5771.745994] LustreError: 422636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.756630] LustreError: 422636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5772.294471] LustreError: 422617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933425 with bad export cookie 1374335072483022004 [ 5772.301961] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.302154] LustreError: 422617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5776.865125] LustreError: 422655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.866255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.874511] LustreError: 422655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5776.884039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.888814] LustreError: 422636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.899436] LustreError: 422636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5778.411187] Lustre: server umount lustre-OST0000 complete [ 5783.007304] LustreError: 423367:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e6121c33 x1869541152941696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5783.064875] Lustre: server umount lustre-MDT0001 complete [ 5786.895895] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5788.179768] Key type lgssc unregistered [ 5788.348687] LNet: 426319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5788.353757] LNetError: 426319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5788.362528] LNet: Removed LNI 192.168.204.144@tcp [ 5788.827736] Key type .llcrypt unregistered [ 5788.830146] Key type ._llcrypt unregistered [ 5799.393986] Key type ._llcrypt registered [ 5799.395232] Key type .llcrypt registered [ 5799.449400] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5799.770182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5799.801791] alg: No test for adler32 (adler32-zlib) [ 5800.667497] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5800.766061] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5802.351190] Key type lgssc registered [ 5802.872750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5806.018893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5806.032771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.213500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.878192] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5813.096446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.254499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5814.516580] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5817.746141] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5819.879722] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.119079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.271117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.643541] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5829.038524] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5831.468163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5836.770322] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.771358] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5836.777113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5836.781324] Lustre: Skipped 1 previous similar message [ 5840.542265] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5841.889774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5841.891891] Lustre: Skipped 1 previous similar message [ 5842.339752] Lustre: server umount lustre-MDT0001 complete [ 5847.008901] LustreError: 427730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.010359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.010690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5847.015019] LustreError: 427730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5847.020653] Lustre: Skipped 1 previous similar message [ 5847.021490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5850.058292] Lustre: server umount lustre-OST0000 complete [ 5851.352673] Lustre: server umount lustre-MDT0000 complete [ 5856.413097] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5857.341093] Key type lgssc unregistered [ 5857.460344] LNet: 431395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5857.462632] LNetError: 431395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5857.473257] LNet: Removed LNI 192.168.204.144@tcp [ 5857.774724] Key type .llcrypt unregistered [ 5857.776073] Key type ._llcrypt unregistered [ 5867.544688] Key type ._llcrypt registered [ 5867.546828] Key type .llcrypt registered [ 5867.620274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5874.865285] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5875.373749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5875.381115] alg: No test for adler32 (adler32-zlib) [ 5876.237104] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 5876.331102] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5877.919195] Key type lgssc registered [ 5878.288490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5892.760715] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5899.082358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5899.088468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.212933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5900.225254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5900.278622] Lustre: lustre-MDT0000: new disk, initializing [ 5900.304099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5900.313225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5901.641657] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5908.925848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5909.062746] Lustre: lustre-OST0000: new disk, initializing [ 5909.066486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5909.070735] Lustre: Skipped 1 previous similar message [ 5909.074935] Lustre: 436398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5909.111097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.674175] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5915.657040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5915.661605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5915.675067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5916.198588] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5923.029653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.086894] Lustre: 435382:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.122687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5923.169115] Lustre: lustre-MDT0001: new disk, initializing [ 5923.212127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5923.230097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5923.234684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5924.873992] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5929.746989] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5929.818557] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5931.637605] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5931.758070] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5935.277951] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5936.098233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.098659] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5936.099645] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5936.104937] Lustre: Skipped 1 previous similar message [ 5940.383527] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5940.388345] Lustre: Skipped 1 previous similar message [ 5945.502710] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5945.508239] Lustre: Skipped 2 previous similar messages [ 5948.383157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5948.462130] Lustre: server umount lustre-MDT0001 complete [ 5956.580129] LustreError: 435387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.590043] LustreError: 435387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5961.698401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5961.698560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.698927] LustreError: 435391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.722884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.726458] Lustre: Skipped 2 previous similar messages [ 5966.819723] LustreError: 435392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.467782] Lustre: server umount lustre-OST0000 complete [ 5975.458719] Lustre: server umount lustre-MDT0000 complete [ 5980.559652] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5981.937890] Key type lgssc unregistered [ 5982.083521] LNet: 440145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.087274] LNetError: 440145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.096597] LNet: Removed LNI 192.168.204.144@tcp [ 5982.500294] Key type .llcrypt unregistered [ 5982.502908] Key type ._llcrypt unregistered [ 5993.929964] Key type ._llcrypt registered [ 5993.931656] Key type .llcrypt registered [ 5993.965960] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5994.052192] Key type .llcrypt unregistered [ 5994.053663] Key type ._llcrypt unregistered [ 6000.069394] Key type ._llcrypt registered [ 6000.070475] Key type .llcrypt registered [ 6000.101994] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6000.399732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6000.440137] alg: No test for adler32 (adler32-zlib) [ 6001.278578] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 6001.359650] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6002.935112] Key type lgssc registered [ 6003.239674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6013.703595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6013.715688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.822376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6014.837478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6014.888471] Lustre: lustre-MDT0000: new disk, initializing [ 6014.920090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.934086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6016.685722] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6021.457937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.529135] Lustre: 443491:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6021.569751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6021.573589] Lustre: Skipped 1 previous similar message [ 6021.614951] Lustre: lustre-MDT0001: new disk, initializing [ 6021.656742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6021.676950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6021.683596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6023.437246] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6028.306408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6030.514171] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6033.627160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.771225] Lustre: lustre-OST0000: new disk, initializing [ 6033.773559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6033.776906] Lustre: 445460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6033.811049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.954734] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6039.041917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6039.049286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6039.084634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6040.435392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6044.131286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.134629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6044.140598] Lustre: Skipped 2 previous similar messages [ 6044.144854] Lustre: Skipped 1 previous similar message [ 6049.280446] Lustre: server umount lustre-MDT0000 complete [ 6051.103971] LustreError: 443486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933704 with bad export cookie 18056795454511293981 [ 6051.109838] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.111816] LustreError: 443486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6051.268424] Lustre: server umount lustre-MDT0001 complete [ 6063.057052] Lustre: server umount lustre-OST0000 complete [ 6067.847856] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:22:01 (1782933721) [ 6070.881634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.093055] LustreError: 447484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.107405] LustreError: 447484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6071.118515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.876169] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6076.359709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.384778] LustreError: 447485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.492507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6077.862172] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6080.673963] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6082.958202] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6098.659918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.756520] Lustre: Found index 0 for lustre-OST0000, updating log [ 6098.843929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.570137] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6104.704231] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6117.292948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6122.470226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.470803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.473181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.473189] Lustre: Skipped 1 previous similar message [ 6122.478601] Lustre: Skipped 1 previous similar message [ 6122.483978] LustreError: Skipped 1 previous similar message [ 6123.553835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.558143] Lustre: Skipped 2 previous similar messages [ 6124.060864] Lustre: server umount lustre-OST0000 complete [ 6137.824090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6137.827824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.833770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.874748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.149266] Lustre: server umount lustre-MDT0000 complete [ 6144.897989] LustreError: 449326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782933798 with bad export cookie 18056795454511295374 [ 6144.904349] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.904779] LustreError: 449326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6145.053143] Lustre: server umount lustre-MDT0001 complete [ 6148.899810] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:23:22 (1782933802) [ 6162.143242] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [ 6732.412567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.632376] LustreError: 453907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.668391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.307607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6737.657080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.847296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6739.681489] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6742.609476] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6744.898895] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6761.188311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.313489] Lustre: Found index 0 for lustre-OST0000, updating log [ 6761.447235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6763.728122] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6765.883175] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6769.639727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6769.650739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6769.651579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6769.659490] Lustre: Skipped 2 previous similar messages [ 6772.649711] Lustre: server umount lustre-OST0000 complete [ 6775.074512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.076553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6775.081846] Lustre: Skipped 1 previous similar message [ 6775.088018] Lustre: Skipped 1 previous similar message [ 6776.096497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.099346] Lustre: Skipped 1 previous similar message [ 6778.145136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6778.148279] Lustre: Skipped 3 previous similar messages [ 6780.193951] LustreError: 453908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.201669] LustreError: 453908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6780.208744] Lustre: server umount lustre-MDT0000 complete [ 6781.217532] LustreError: 454635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.234568] LustreError: 454635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6781.632981] LustreError: 453889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782934435 with bad export cookie 18056795454511296445 [ 6781.638883] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.640179] LustreError: 453889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6781.782998] Lustre: server umount lustre-MDT0001 complete [ 6790.879303] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [ 6800.619842] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6805.023028] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6820.744634] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:34:33 (1782934473) [ 6828.207789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6829.623297] Key type lgssc unregistered [ 6829.788461] LNet: 460562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6829.792088] LNetError: 460562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6891.237303] LNet: Removed LNI 192.168.204.144@tcp [ 6891.524646] Key type .llcrypt unregistered [ 6891.525902] Key type ._llcrypt unregistered [ 6899.093524] Key type ._llcrypt registered [ 6899.095201] Key type .llcrypt registered [ 6899.149460] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6899.450476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6899.483450] alg: No test for adler32 (adler32-zlib) [ 6900.346766] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 6900.456503] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6902.055185] Key type lgssc registered [ 6902.519260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6904.432845] LNet: Added LNI 192.168.204.144@tcp100 [8/256/0/180] [ 6904.819982] LNet: Added LNI 192.168.204.144@tcp101 [8/256/0/180] [ 6905.242768] LNet: Added LNI 192.168.204.144@tcp102 [8/256/0/180] [ 6905.674130] LNet: Added LNI 192.168.204.144@tcp103 [8/256/0/180] [ 6906.115051] LNet: Added LNI 192.168.204.144@tcp104 [8/256/0/180] [ 6906.546803] LNet: Added LNI 192.168.204.144@tcp105 [8/256/0/180] [ 6906.972535] LNet: Added LNI 192.168.204.144@tcp106 [8/256/0/180] [ 6907.416294] LNet: Added LNI 192.168.204.144@tcp107 [8/256/0/180] [ 6907.863724] LNet: Added LNI 192.168.204.144@tcp108 [8/256/0/180] [ 6908.271815] LNet: Added LNI 192.168.204.144@tcp109 [8/256/0/180] [ 6908.691224] LNet: Added LNI 192.168.204.144@tcp110 [8/256/0/180] [ 6909.112196] LNet: Added LNI 192.168.204.144@tcp111 [8/256/0/180] [ 6909.529775] LNet: Added LNI 192.168.204.144@tcp112 [8/256/0/180] [ 6909.977454] LNet: Added LNI 192.168.204.144@tcp113 [8/256/0/180] [ 6910.376975] LNet: Added LNI 192.168.204.144@tcp114 [8/256/0/180] [ 6910.827800] LNet: Added LNI 192.168.204.144@tcp115 [8/256/0/180] [ 6911.247800] LNet: Added LNI 192.168.204.144@tcp116 [8/256/0/180] [ 6911.638848] LNet: Added LNI 192.168.204.144@tcp117 [8/256/0/180] [ 6912.026798] LNet: Added LNI 192.168.204.144@tcp118 [8/256/0/180] [ 6912.469727] LNet: Added LNI 192.168.204.144@tcp119 [8/256/0/180] [ 6912.914512] LNet: Added LNI 192.168.204.144@tcp120 [8/256/0/180] [ 6913.340366] LNet: Added LNI 192.168.204.144@tcp121 [8/256/0/180] [ 6913.768920] LNet: Added LNI 192.168.204.144@tcp122 [8/256/0/180] [ 6914.183571] LNet: Added LNI 192.168.204.144@tcp123 [8/256/0/180] [ 6914.606565] LNet: Added LNI 192.168.204.144@tcp124 [8/256/0/180] [ 6915.008671] LNet: Added LNI 192.168.204.144@tcp125 [8/256/0/180] [ 6915.368447] LNet: Added LNI 192.168.204.144@tcp126 [8/256/0/180] [ 6915.778607] LNet: Added LNI 192.168.204.144@tcp127 [8/256/0/180] [ 6916.194475] LNet: Added LNI 192.168.204.144@tcp128 [8/256/0/180] [ 6916.611418] LNet: Added LNI 192.168.204.144@tcp129 [8/256/0/180] [ 6917.038987] LNet: Added LNI 192.168.204.144@tcp130 [8/256/0/180] [ 6917.443292] LNet: Added LNI 192.168.204.144@tcp131 [8/256/0/180] [ 6917.863160] LNet: Added LNI 192.168.204.144@tcp132 [8/256/0/180] [ 6918.298144] LNet: Added LNI 192.168.204.144@tcp133 [8/256/0/180] [ 6918.723690] LNet: Added LNI 192.168.204.144@tcp134 [8/256/0/180] [ 6919.146112] LNet: Added LNI 192.168.204.144@tcp135 [8/256/0/180] [ 6922.336604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6922.348083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.397607] Lustre: 463559:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 6923.428668] LNetError: 463579:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6923.431404] LNetError: Protocol error connecting to 192.168.204.144@tcp119 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6923.434048] LNetError: Skipped 4 previous similar messages [ 6923.469487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6923.481604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6923.516553] Lustre: lustre-MDT0000: new disk, initializing [ 6923.538207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.543986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6923.999136] LNet: 8 local NIs in recovery (showing 8): 192.168.204.144@tcp122, 192.168.204.144@tcp123, 192.168.204.144@tcp124, 192.168.204.144@tcp125, 192.168.204.144@tcp119, 192.168.204.144@tcp127, 192.168.204.144@tcp129, 192.168.204.144@tcp126 [ 6924.450302] LNetError: 463581:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6924.455106] LNetError: 463581:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 6924.458576] LNetError: Protocol error connecting to 192.168.204.144@tcp104 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6924.462522] LNetError: Skipped 7 previous similar messages [ 6925.267923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6927.078350] LNetError: 463585:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6927.089314] LNetError: 463585:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 6927.097674] LNetError: Protocol error connecting to 192.168.204.144@tcp122 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6927.106277] LNetError: Skipped 10 previous similar messages [ 6930.647372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.731181] Lustre: 463576:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.760656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6930.764439] Lustre: Skipped 1 previous similar message [ 6930.804307] Lustre: lustre-MDT0001: new disk, initializing [ 6930.817169] Lustre: 464378:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 6930.820542] Lustre: 464378:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6930.856743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.870225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.873376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6931.168676] LNetError: 463584:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6931.176702] LNetError: 463584:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6931.182177] LNetError: Protocol error connecting to 192.168.204.144@tcp122 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6932.557878] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6937.327601] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6937.585657] Lustre: 465000:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 6937.589215] Lustre: 465000:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6939.360894] LNetError: 461021:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6939.370239] LNetError: Protocol error connecting to 192.168.204.144@tcp122 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6939.751789] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6942.607118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.773335] Lustre: lustre-OST0000: new disk, initializing [ 6942.776484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6942.781209] Lustre: 465552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6942.829228] Lustre: 465557:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 6942.837427] Lustre: 465557:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6942.867550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.324966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6947.835334] Lustre: 466069:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 6947.842291] Lustre: 466069:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6947.867971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6947.871352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6947.923940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6949.642970] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6952.931591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.933442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.935421] LustreError: Skipped 1 previous similar message [ 6952.943312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6952.948871] Lustre: Skipped 1 previous similar message [ 6954.988958] Lustre: server umount lustre-OST0000 complete [ 6955.744620] LNetError: 463580:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6955.750759] LNetError: Protocol error connecting to 192.168.204.144@tcp122 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6958.049347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.050575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6958.055049] Lustre: Skipped 2 previous similar messages [ 6958.061015] Lustre: Skipped 1 previous similar message [ 6963.169400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.173818] Lustre: Skipped 1 previous similar message [ 6968.289813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6968.295609] Lustre: Skipped 1 previous similar message [ 6971.359258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.456842] Lustre: server umount lustre-MDT0000 complete [ 6973.150086] LustreError: 463568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782934626 with bad export cookie 17547184916688670536 [ 6973.153413] LNetError: 463579:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [ 6973.154313] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.157862] LustreError: 463568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6973.163967] LNetError: 463579:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 6973.164108] LNetError: Protocol error connecting to 192.168.204.144@tcp128 at host 192.168.204.144:988: is it running a compatible version of Lustre? [ 6973.185558] LNetError: Skipped 11 previous similar messages [ 6973.299482] Lustre: server umount lustre-MDT0001 complete [ 6977.393235] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6978.472700] Key type lgssc unregistered [ 6978.688566] LNet: 467463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6978.694734] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6978.699275] LNet: Removed LNI 192.168.204.144@tcp [ 6979.743271] LNet: Removed LNI 192.168.204.144@tcp100 [ 6979.747570] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6979.754735] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6979.762309] LNet: Removed LNI 192.168.204.144@tcp101 [ 6980.767188] LNet: Removed LNI 192.168.204.144@tcp102 [ 6980.770841] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6980.778144] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6980.785303] LNet: Removed LNI 192.168.204.144@tcp103 [ 6981.791204] LNet: Removed LNI 192.168.204.144@tcp104 [ 6982.815226] LNet: Removed LNI 192.168.204.144@tcp105 [ 6982.817693] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6982.822154] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6982.827212] LNet: Removed LNI 192.168.204.144@tcp106 [ 6983.839171] LNet: Removed LNI 192.168.204.144@tcp107 [ 6984.863183] LNet: Removed LNI 192.168.204.144@tcp108 [ 6985.887197] LNet: Removed LNI 192.168.204.144@tcp109 [ 6986.911162] LNet: Removed LNI 192.168.204.144@tcp110 [ 6986.913238] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6986.917069] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6986.921369] LNet: Removed LNI 192.168.204.144@tcp111 [ 6987.935188] LNet: Removed LNI 192.168.204.144@tcp112 [ 6988.959183] LNet: Removed LNI 192.168.204.144@tcp113 [ 6989.983196] LNet: Removed LNI 192.168.204.144@tcp114 [ 6991.007256] LNet: Removed LNI 192.168.204.144@tcp115 [ 6992.031161] LNet: Removed LNI 192.168.204.144@tcp116 [ 6993.055099] LNet: Removed LNI 192.168.204.144@tcp117 [ 6994.079096] LNet: Removed LNI 192.168.204.144@tcp118 [ 6995.103127] LNet: Removed LNI 192.168.204.144@tcp119 [ 6995.105052] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6995.107138] LNetError: 467463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 6995.111043] LNet: Removed LNI 192.168.204.144@tcp120 [ 6996.127117] LNet: Removed LNI 192.168.204.144@tcp121 [ 6996.129582] LNet: Removed LNI 192.168.204.144@tcp122 [ 6997.151218] LNet: Removed LNI 192.168.204.144@tcp123 [ 6998.175181] LNet: Removed LNI 192.168.204.144@tcp124 [ 6999.199257] LNet: Removed LNI 192.168.204.144@tcp125 [ 7000.223122] LNet: Removed LNI 192.168.204.144@tcp126 [ 7001.247207] LNet: Removed LNI 192.168.204.144@tcp127 [ 7002.271119] LNet: Removed LNI 192.168.204.144@tcp128 [ 7003.295172] LNet: Removed LNI 192.168.204.144@tcp129 [ 7003.298300] LNet: Removed LNI 192.168.204.144@tcp130 [ 7003.301330] LNet: Removed LNI 192.168.204.144@tcp131 [ 7003.304124] LNet: Removed LNI 192.168.204.144@tcp132 [ 7003.307034] LNet: Removed LNI 192.168.204.144@tcp133 [ 7003.310031] LNet: Removed LNI 192.168.204.144@tcp134 [ 7003.319790] LNet: Removed LNI 192.168.204.144@tcp135 [ 7003.755555] Key type .llcrypt unregistered [ 7003.758663] Key type ._llcrypt unregistered [ 7014.518823] Key type ._llcrypt registered [ 7014.520639] Key type .llcrypt registered [ 7014.584809] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7022.692616] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7023.100249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7023.127139] alg: No test for adler32 (adler32-zlib) [ 7024.009846] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 7024.136859] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7025.743200] Key type lgssc registered [ 7026.342260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7043.400422] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:38:16 (1782934696) [ 7051.422599] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7052.918067] Key type lgssc unregistered [ 7053.097706] LNet: 471524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7053.104740] LNetError: 471524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7053.119554] LNet: Removed LNI 192.168.204.144@tcp [ 7053.596079] Key type .llcrypt unregistered [ 7053.598596] Key type ._llcrypt unregistered [ 7061.821548] Key type ._llcrypt registered [ 7061.822746] Key type .llcrypt registered [ 7061.869929] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7062.287144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7062.299529] alg: No test for adler32 (adler32-zlib) [ 7063.182297] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 7063.310725] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7064.919254] Key type lgssc registered [ 7065.395960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.318608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7069.325392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.413183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7070.422259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7070.461825] Lustre: lustre-MDT0000: new disk, initializing [ 7070.486405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.495794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7072.184931] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7076.223258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.260034] Lustre: 472724:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7076.280205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7076.282338] Lustre: Skipped 1 previous similar message [ 7076.320024] Lustre: lustre-MDT0001: new disk, initializing [ 7076.351324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7076.364096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7076.366767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7077.508690] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7080.860176] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7082.396907] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7084.323938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.419199] Lustre: lustre-OST0000: new disk, initializing [ 7084.420721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7084.422860] Lustre: 474689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7084.446901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.059147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7085.063905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7085.097714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7086.219734] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7089.738215] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7090.885018] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lnet_if_list [ 7092.393894] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7092.403327] LNet: Added LNI 192.168.204.144@tcp10 [8/256/0/180] [ 7094.023445] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7094.034310] LNet: Added LNI 192.168.204.144@tcp11 [8/256/0/180] [ 7095.177917] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7095.187419] LNet: Added LNI 192.168.204.144@tcp12 [8/256/0/180] [ 7096.469541] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7096.479761] LNet: Added LNI 192.168.204.144@tcp13 [8/256/0/180] [ 7097.764018] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7097.774949] LNet: Added LNI 192.168.204.144@tcp14 [8/256/0/180] [ 7099.036298] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7099.045315] LNet: Added LNI 192.168.204.144@tcp15 [8/256/0/180] [ 7100.389073] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7100.399070] LNet: Added LNI 192.168.204.144@tcp16 [8/256/0/180] [ 7101.787472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7101.796924] LNet: Added LNI 192.168.204.144@tcp17 [8/256/0/180] [ 7103.277019] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7103.287184] LNet: Added LNI 192.168.204.144@tcp18 [8/256/0/180] [ 7104.638657] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7104.648273] LNet: Added LNI 192.168.204.144@tcp19 [8/256/0/180] [ 7106.005721] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7106.016639] LNet: Added LNI 192.168.204.144@tcp20 [8/256/0/180] [ 7107.378501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7107.388950] LNet: Added LNI 192.168.204.144@tcp21 [8/256/0/180] [ 7108.782803] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7108.793351] LNet: Added LNI 192.168.204.144@tcp22 [8/256/0/180] [ 7110.253491] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7110.266162] LNet: Added LNI 192.168.204.144@tcp23 [8/256/0/180] [ 7111.834101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7111.843370] LNet: Added LNI 192.168.204.144@tcp24 [8/256/0/180] [ 7113.482852] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7113.492618] LNet: Added LNI 192.168.204.144@tcp25 [8/256/0/180] [ 7115.258274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7115.268239] LNet: Added LNI 192.168.204.144@tcp26 [8/256/0/180] [ 7116.985113] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7116.994945] LNet: Added LNI 192.168.204.144@tcp27 [8/256/0/180] [ 7118.811750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7118.820928] LNet: Added LNI 192.168.204.144@tcp28 [8/256/0/180] [ 7120.830196] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7120.840793] LNet: Added LNI 192.168.204.144@tcp29 [8/256/0/180] [ 7123.708163] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7123.721120] LNet: Added LNI 192.168.204.144@tcp30 [8/256/0/180] [ 7126.264851] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7126.280323] LNet: Added LNI 192.168.204.144@tcp31 [8/256/0/180] [ 7128.892108] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7128.904450] LNet: Added LNI 192.168.204.144@tcp32 [8/256/0/180] [ 7131.283437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7131.296957] LNet: Added LNI 192.168.204.144@tcp33 [8/256/0/180] [ 7133.711795] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7133.727460] LNet: Added LNI 192.168.204.144@tcp34 [8/256/0/180] [ 7136.721575] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7136.731902] LNet: Added LNI 192.168.204.144@tcp35 [8/256/0/180] [ 7139.109912] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7139.119709] LNet: Added LNI 192.168.204.144@tcp36 [8/256/0/180] [ 7141.272550] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7141.282073] LNet: Added LNI 192.168.204.144@tcp37 [8/256/0/180] [ 7143.498479] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7143.508441] LNet: Added LNI 192.168.204.144@tcp38 [8/256/0/180] [ 7145.662959] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7145.679876] LNet: Added LNI 192.168.204.144@tcp39 [8/256/0/180] [ 7148.881525] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7148.900911] LNet: Added LNI 192.168.204.144@tcp40 [8/256/0/180] [ 7152.243678] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7152.260753] LNet: Added LNI 192.168.204.144@tcp41 [8/256/0/180] [ 7155.682007] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7155.700169] LNet: Added LNI 192.168.204.144@tcp42 [8/256/0/180] [ 7159.229840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7159.246762] LNet: Added LNI 192.168.204.144@tcp43 [8/256/0/180] [ 7162.706350] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7162.725616] LNet: Added LNI 192.168.204.144@tcp44 [8/256/0/180] [ 7165.598887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7165.612956] LNet: Added LNI 192.168.204.144@tcp45 [8/256/0/180] [ 7169.468483] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7169.488181] LNet: Added LNI 192.168.204.144@tcp46 [8/256/0/180] [ 7173.706360] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7173.727641] LNet: Added LNI 192.168.204.144@tcp47 [8/256/0/180] [ 7177.244856] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7177.258628] LNet: Added LNI 192.168.204.144@tcp48 [8/256/0/180] [ 7180.667462] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7180.688622] LNet: Added LNI 192.168.204.144@tcp49 [8/256/0/180] [ 7184.322897] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7184.335794] LNet: Added LNI 192.168.204.144@tcp50 [8/256/0/180] [ 7194.080595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7194.086168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7194.095845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7197.666579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7197.670302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7199.237756] Lustre: server umount lustre-MDT0000 complete [ 7201.062283] LustreError: 472715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782934854 with bad export cookie 8232272088559186063 [ 7201.068759] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7201.223598] Lustre: server umount lustre-MDT0001 complete [ 7209.742841] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7213.517536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.103074] Lustre: 471998:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782934856/real 1782934856] req@0000000028f4f6e5 x1869542552852096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782934872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7219.109227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.112276] Lustre: Skipped 1 previous similar message [ 7223.199136] Lustre: 471998:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782934860/real 1782934860] req@0000000023bc3cc6 x1869542552852224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782934876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7224.287099] Lustre: 471996:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782934861/real 1782934861] req@00000000f4579fdc x1869542552852352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782934877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7226.271555] LustreError: 471995:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f49977f x1869542552853248/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7226.335153] Lustre: 471997:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782934864/real 1782934864] req@0000000093f067b9 x1869542552852608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782934880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7226.452155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7228.136322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7231.583227] Lustre: 471998:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782934869/real 1782934869] req@00000000f6e37716 x1869542552852736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782934885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7232.181062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7232.367427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7234.248064] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7237.171350] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7237.474493] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7239.042334] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7247.841796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.847904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.858192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.861220] Lustre: Skipped 3 previous similar messages [ 7252.965400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.971952] Lustre: Skipped 1 previous similar message [ 7253.175776] Lustre: server umount lustre-OST0000 complete [ 7258.082127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7258.082729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.084093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.098371] Lustre: Skipped 2 previous similar messages [ 7260.783900] Lustre: server umount lustre-MDT0000 complete [ 7262.589059] LustreError: 500462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782934916 with bad export cookie 8232272088559188114 [ 7262.593828] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7262.596165] LustreError: 500462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7262.739519] Lustre: server umount lustre-MDT0001 complete [ 7266.929412] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7268.447634] Key type lgssc unregistered [ 7268.629909] LNet: 501654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7268.632432] LNetError: 501654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7268.634315] LNet: Removed LNI 192.168.204.144@tcp [ 7268.635658] LNet: Removed LNI 192.168.204.144@tcp10 [ 7268.636672] LNet: Removed LNI 192.168.204.144@tcp11 [ 7268.637608] LNet: Removed LNI 192.168.204.144@tcp12 [ 7268.638591] LNet: Removed LNI 192.168.204.144@tcp13 [ 7268.639562] LNet: Removed LNI 192.168.204.144@tcp14 [ 7268.640536] LNet: Removed LNI 192.168.204.144@tcp15 [ 7268.641490] LNet: Removed LNI 192.168.204.144@tcp16 [ 7268.642476] LNet: Removed LNI 192.168.204.144@tcp17 [ 7268.643419] LNet: Removed LNI 192.168.204.144@tcp18 [ 7268.644484] LNet: Removed LNI 192.168.204.144@tcp19 [ 7268.645522] LNet: Removed LNI 192.168.204.144@tcp20 [ 7268.646505] LNet: Removed LNI 192.168.204.144@tcp21 [ 7268.647438] LNet: Removed LNI 192.168.204.144@tcp22 [ 7268.648383] LNet: Removed LNI 192.168.204.144@tcp23 [ 7268.649347] LNet: Removed LNI 192.168.204.144@tcp24 [ 7268.650299] LNet: Removed LNI 192.168.204.144@tcp25 [ 7268.651270] LNet: Removed LNI 192.168.204.144@tcp26 [ 7268.652609] LNet: Removed LNI 192.168.204.144@tcp27 [ 7268.653624] LNet: Removed LNI 192.168.204.144@tcp28 [ 7268.654972] LNet: Removed LNI 192.168.204.144@tcp29 [ 7268.656081] LNet: Removed LNI 192.168.204.144@tcp30 [ 7268.657126] LNet: Removed LNI 192.168.204.144@tcp31 [ 7268.658141] LNet: Removed LNI 192.168.204.144@tcp32 [ 7268.659206] LNet: Removed LNI 192.168.204.144@tcp33 [ 7268.660436] LNet: Removed LNI 192.168.204.144@tcp34 [ 7268.661483] LNet: Removed LNI 192.168.204.144@tcp35 [ 7268.662469] LNet: Removed LNI 192.168.204.144@tcp36 [ 7268.663450] LNet: Removed LNI 192.168.204.144@tcp37 [ 7268.664384] LNet: Removed LNI 192.168.204.144@tcp38 [ 7268.665369] LNet: Removed LNI 192.168.204.144@tcp39 [ 7268.666322] LNet: Removed LNI 192.168.204.144@tcp40 [ 7268.667271] LNet: Removed LNI 192.168.204.144@tcp41 [ 7268.668219] LNet: Removed LNI 192.168.204.144@tcp42 [ 7268.669172] LNet: Removed LNI 192.168.204.144@tcp43 [ 7268.670141] LNet: Removed LNI 192.168.204.144@tcp44 [ 7268.671162] LNet: Removed LNI 192.168.204.144@tcp45 [ 7268.672164] LNet: Removed LNI 192.168.204.144@tcp46 [ 7268.673212] LNet: Removed LNI 192.168.204.144@tcp47 [ 7268.674152] LNet: Removed LNI 192.168.204.144@tcp48 [ 7268.675157] LNet: Removed LNI 192.168.204.144@tcp49 [ 7268.685339] LNet: Removed LNI 192.168.204.144@tcp50 [ 7268.929680] Key type .llcrypt unregistered [ 7268.930958] Key type ._llcrypt unregistered [ 7277.698570] Key type ._llcrypt registered [ 7277.700288] Key type .llcrypt registered [ 7277.749905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7284.360201] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7284.673632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7284.702932] alg: No test for adler32 (adler32-zlib) [ 7285.561194] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 7285.649789] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7287.239190] Key type lgssc registered [ 7287.751850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7302.090142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7302.095861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.218145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.228909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7303.268387] Lustre: lustre-MDT0000: new disk, initializing [ 7303.291944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.299712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7304.949512] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7309.980260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.015502] Lustre: 504822:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7310.035064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7310.036532] Lustre: Skipped 1 previous similar message [ 7310.069661] Lustre: lustre-MDT0001: new disk, initializing [ 7310.097569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.107458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7310.110110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7311.231270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7315.863554] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7318.325366] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7321.316103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.422848] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7321.462154] Lustre: lustre-OST0000: new disk, initializing [ 7321.464590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7321.468113] Lustre: 506838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7321.506197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7323.453953] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7327.571731] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7327.748035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7327.752765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7327.791277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7332.834257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7332.837312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.838137] LustreError: Skipped 1 previous similar message [ 7332.849104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7332.852523] Lustre: Skipped 1 previous similar message [ 7336.808068] Lustre: server umount lustre-OST0000 complete [ 7341.024788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7341.029990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.037676] Lustre: Skipped 1 previous similar message [ 7341.041417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.073448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.074780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.826249] Lustre: server umount lustre-MDT0000 complete [ 7346.290643] LustreError: 506837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782934999 with bad export cookie 17735685543219117499 [ 7346.295762] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.296278] LustreError: 506837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7346.436491] Lustre: server umount lustre-MDT0001 complete [ 7350.074489] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7351.428217] Key type lgssc unregistered [ 7351.589675] LNet: 508745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.594178] LNetError: 508745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.607582] LNet: Removed LNI 192.168.204.144@tcp [ 7352.100151] Key type .llcrypt unregistered [ 7352.102414] Key type ._llcrypt unregistered [ 7360.612606] Key type ._llcrypt registered [ 7360.614237] Key type .llcrypt registered [ 7360.674536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7361.124573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7361.134230] alg: No test for adler32 (adler32-zlib) [ 7361.993634] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 7362.116130] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7363.727163] Key type lgssc registered [ 7364.359318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7367.405451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7367.411514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.609932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7370.222764] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7373.850846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.990607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7375.579259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7378.231053] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7380.575206] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7383.916778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.131397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.008227] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7390.513799] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7400.451480] loop0: detected capacity change from 0 to 204800000 [ 7400.588388] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7400.594198] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7400.599683] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7403.538827] loop0: detected capacity change from 0 to 204800000 [ 7403.644546] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7403.648845] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7403.652520] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7406.036136] loop0: detected capacity change from 0 to 204800000 [ 7408.737778] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.821888] Lustre: fs15246-OST0000: new disk, initializing [ 7408.825565] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7408.831244] Lustre: 513585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7408.866782] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.800657] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7417.079746] loop1: detected capacity change from 0 to 204800000 [ 7419.847756] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.921540] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7419.964587] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7420.010435] Lustre: fs15246-MDT0000: new disk, initializing [ 7420.091678] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7420.104698] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7420.156390] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7421.216164] Lustre: 514708:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000065d6ec92 x1869542866323328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782935090 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7422.041758] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7430.307096] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7430.313439] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7430.328937] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7433.705392] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.714902] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.061779] Lustre: server umount fs15246-MDT0000 complete [ 7438.824147] LustreError: 509987:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.834556] LustreError: 509987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7442.478805] Lustre: server umount fs15246-OST0000 complete [ 7449.060342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.062477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7449.068452] Lustre: Skipped 1 previous similar message [ 7449.075086] Lustre: Skipped 1 previous similar message [ 7453.344436] Lustre: server umount lustre-OST0000 complete [ 7456.224412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7456.229536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.238244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.297141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.298305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7461.298038] Lustre: server umount lustre-MDT0000 complete [ 7463.102669] LustreError: 509971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782935116 with bad export cookie 16325967803032051252 [ 7463.106863] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.109069] LustreError: 509971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7463.242240] Lustre: server umount lustre-MDT0001 complete [ 7467.399312] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7468.741888] Key type lgssc unregistered [ 7468.899749] LNet: 517801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7468.904874] LNetError: 517801:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7468.915662] LNet: Removed LNI 192.168.204.144@tcp [ 7469.333155] Key type .llcrypt unregistered [ 7469.334448] Key type ._llcrypt unregistered [ 7477.346879] Key type ._llcrypt registered [ 7477.348329] Key type .llcrypt registered [ 7477.404563] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7477.901187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7477.921809] alg: No test for adler32 (adler32-zlib) [ 7478.816267] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 7478.936252] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7480.535157] Key type lgssc registered [ 7481.087313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7499.017385] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7503.174056] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7519.434705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7519.447313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.580990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7520.600292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7520.652918] Lustre: lustre-MDT0000: new disk, initializing [ 7520.690549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.700641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7522.524801] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7528.120867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.203781] Lustre: 522237:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7528.250425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7528.254025] Lustre: Skipped 1 previous similar message [ 7528.304393] Lustre: lustre-MDT0001: new disk, initializing [ 7528.362461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.385805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7528.393092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7530.260517] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7535.054684] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7537.207699] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7540.474070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.622077] Lustre: lustre-OST0000: new disk, initializing [ 7540.625314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7540.630505] Lustre: 524205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7540.675911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.583062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7541.588473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7541.631166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7543.372729] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7548.123448] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7549.944257] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7550.063566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7552.082707] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7552.195773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7557.096955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.097700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.107958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.116227] Lustre: Skipped 1 previous similar message [ 7559.283385] Lustre: server umount lustre-OST0000 complete [ 7562.211223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7562.212900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7562.219404] Lustre: Skipped 2 previous similar messages [ 7567.289601] Lustre: server umount lustre-MDT0000 complete [ 7569.118488] LustreError: 523031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782935222 with bad export cookie 7080221063265298623 [ 7569.125615] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.126111] LustreError: 523031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7569.309219] Lustre: server umount lustre-MDT0001 complete [ 7580.753692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7585.174772] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7599.834298] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7456 sec ========== 15:47:32 (1782935252) [ 7600.662771] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:47:33 (1782935253) === [ 7602.204434] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:47:35 (1782935255) === [ 7619.797286] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7620.917755] Key type lgssc unregistered [ 7621.104734] LNet: 530840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7621.109901] LNetError: 530840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7621.122737] LNet: Removed LNI 192.168.204.144@tcp [ 7621.614497] Key type .llcrypt unregistered [ 7621.616671] Key type ._llcrypt unregistered