[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516659492 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011045] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.021230] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027019] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029078] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030104] *** VALIDATE tmpfs *** [ 0.031412] *** VALIDATE proc *** [ 0.032217] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041281] debug: unmapping init [mem 0xffffffffa6a59000-0xffffffffa6a60fff] [ 0.043168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044772] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045037] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052360] rcu: Hierarchical SRCU implementation. [ 0.055437] smp: Bringing up secondary CPUs ... [ 0.056716] x86: Booting SMP configuration: [ 0.057041] .... node #0, CPUs: #1 #2 #3 [ 0.060404] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145391] node 0 deferred pages initialised in 81ms [ 0.149199] devtmpfs: initialized [ 0.150257] x86/mm: Memory block size: 128MB [ 0.154135] gcov: version magic: 0x41383552 [ 0.158321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163358] pinctrl core: initialized pinctrl subsystem [ 0.165165] [ 0.165831] ************************************************************* [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171013] ** ** [ 0.174013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176009] ** ** [ 0.177007] ** This means that this kernel is built to expose internal ** [ 0.178010] ** IOMMU data structures, which may compromise security on ** [ 0.181013] ** your system. ** [ 0.182008] ** ** [ 0.184012] ** If you see this message and you are not debugging the ** [ 0.185007] ** kernel, report this immediately to your vendor! ** [ 0.187012] ** ** [ 0.189009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191008] ************************************************************* [ 0.192612] NET: Registered protocol family 16 [ 0.194358] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203105] cpuidle: using governor menu [ 0.204818] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206333] PCI: Using configuration type 1 for base access [ 0.208115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214163] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216128] cryptd: max_cpu_qlen set to 1000 [ 0.218195] ACPI: Added _OSI(Module Device) [ 0.219009] ACPI: Added _OSI(Processor Device) [ 0.219896] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223915] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228392] ACPI: Interpreter enabled [ 0.229038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229923] ACPI: Using IOAPIC for interrupt routing [ 0.231081] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233254] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241306] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248994] acpiphp: Slot [2] registered [ 0.249056] acpiphp: Slot [5] registered [ 0.249756] acpiphp: Slot [6] registered [ 0.251074] acpiphp: Slot [7] registered [ 0.252019] acpiphp: Slot [8] registered [ 0.252883] acpiphp: Slot [9] registered [ 0.254065] acpiphp: Slot [10] registered [ 0.255051] acpiphp: Slot [3] registered [ 0.255970] acpiphp: Slot [4] registered [ 0.256048] acpiphp: Slot [11] registered [ 0.256955] acpiphp: Slot [12] registered [ 0.258072] acpiphp: Slot [13] registered [ 0.258936] acpiphp: Slot [14] registered [ 0.260051] acpiphp: Slot [15] registered [ 0.260864] acpiphp: Slot [16] registered [ 0.261050] acpiphp: Slot [17] registered [ 0.262052] acpiphp: Slot [18] registered [ 0.262920] acpiphp: Slot [19] registered [ 0.264053] acpiphp: Slot [20] registered [ 0.264938] acpiphp: Slot [21] registered [ 0.266064] acpiphp: Slot [22] registered [ 0.266896] acpiphp: Slot [23] registered [ 0.267048] acpiphp: Slot [24] registered [ 0.267931] acpiphp: Slot [25] registered [ 0.269069] acpiphp: Slot [26] registered [ 0.269982] acpiphp: Slot [27] registered [ 0.271101] acpiphp: Slot [28] registered [ 0.271951] acpiphp: Slot [29] registered [ 0.272063] acpiphp: Slot [30] registered [ 0.272902] acpiphp: Slot [31] registered [ 0.274041] PCI host bridge to bus 0000:00 [ 0.274883] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.284031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286379] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.289240] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298797] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.304050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.312584] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.314697] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.317045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.320212] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.338014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352056] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.391311] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.399020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.414023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.448026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460153] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517477] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.527022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.536079] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.567446] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.580021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.608018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.623961] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.631027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.638031] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667247] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.669391] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.678544] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.681211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685134] iommu: Default domain type: Passthrough [ 0.687781] SCSI subsystem initialized [ 0.689168] ACPI: bus type USB registered [ 0.690160] usbcore: registered new interface driver usbfs [ 0.692126] usbcore: registered new interface driver hub [ 0.694278] usbcore: registered new device driver usb [ 0.696271] pps_core: LinuxPPS API ver. 1 registered [ 0.698017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700116] PTP clock support registered [ 0.702246] EDAC MC: Ver: 3.0.0 [ 0.705178] PCI: Using ACPI for IRQ routing [ 0.706980] NetLabel: Initializing [ 0.708015] NetLabel: domain hash size = 128 [ 0.709012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711125] NetLabel: unlabeled traffic allowed by default [ 0.713651] vgaarb: loaded [ 0.715275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.716011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.720439] clocksource: Switched to clocksource kvm-clock [ 0.842787] VFS: Disk quotas dquot_6.6.0 [ 0.844195] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.846128] *** VALIDATE ramfs *** [ 0.847110] *** VALIDATE hugetlbfs *** [ 0.848513] pnp: PnP ACPI init [ 0.850673] pnp: PnP ACPI: found 6 devices [ 0.867573] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.870237] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871915] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873650] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875479] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877341] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.879711] NET: Registered protocol family 2 [ 0.882787] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.888389] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.892569] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.897861] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.901409] TCP: Hash tables configured (established 65536 bind 65536) [ 0.903917] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907094] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910369] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914481] NET: Registered protocol family 1 [ 0.916791] RPC: Registered named UNIX socket transport module. [ 0.918395] RPC: Registered udp transport module. [ 0.919522] RPC: Registered tcp transport module. [ 0.921423] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924647] NET: Registered protocol family 44 [ 0.931221] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.934866] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937887] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.940754] PCI: CLS 0 bytes, default 64 [ 0.942300] Unpacking initramfs... [ 2.335884] debug: unmapping init [mem 0xffff9bf13cc54000-0xffff9bf13ffbffff] [ 2.338862] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.340424] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.342230] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.835120] Initialise system trusted keyrings [ 2.836744] Key type blacklist registered [ 2.838451] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.848764] zbud: loaded [ 2.852516] *** VALIDATE nfs *** [ 2.854085] *** VALIDATE nfs4 *** [ 2.855721] pstore: using deflate compression [ 2.859973] Platform Keyring initialized [ 2.976625] NET: Registered protocol family 38 [ 2.978607] Key type asymmetric registered [ 2.980545] Asymmetric key parser 'x509' registered [ 2.982533] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.986221] io scheduler mq-deadline registered [ 2.988243] io scheduler kyber registered [ 2.990209] io scheduler bfq registered [ 2.992342] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.995605] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.998264] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.001656] ACPI: Power Button [PWRF] [ 3.006857] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.014101] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.029110] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.036322] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.051566] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.079234] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.108947] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.114234] Non-volatile memory driver v1.3 [ 3.116225] Linux agpgart interface v0.103 [ 3.153335] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.156551] vda: detected capacity change from 0 to 74760192 [ 3.173534] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.177172] vdb: detected capacity change from 0 to 1073741824 [ 3.195033] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.198415] vdc: detected capacity change from 0 to 2621440000 [ 3.212569] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.215504] vdd: detected capacity change from 0 to 2621440000 [ 3.234676] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.238405] vde: detected capacity change from 0 to 4294967296 [ 3.256668] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.260170] vdf: detected capacity change from 0 to 4294967296 [ 3.267332] libphy: Fixed MDIO Bus: probed [ 3.276758] usbcore: registered new interface driver usbserial_generic [ 3.279467] usbserial: USB Serial support registered for generic [ 3.284754] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.289593] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.293393] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.299693] mousedev: PS/2 mouse device common for all mice [ 3.303481] rtc_cmos 00:05: RTC can wake from S4 [ 3.305442] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.308676] rtc_cmos 00:05: registered as rtc0 [ 3.312422] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.315533] intel_pstate: CPU model not supported [ 3.318050] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.320755] hid: raw HID events driver (C) Jiri Kosina [ 3.323178] usbcore: registered new interface driver usbhid [ 3.324323] usbhid: USB HID core driver [ 3.325351] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.325357] drop_monitor: Initializing network drop monitor service [ 3.329629] Initializing XFRM netlink socket [ 3.330794] NET: Registered protocol family 10 [ 3.332752] Segment Routing with IPv6 [ 3.333658] NET: Registered protocol family 17 [ 3.334815] mpls_gso: MPLS GSO support [ 3.338755] RAS: Correctable Errors collector initialized. [ 3.339937] AVX version of gcm_enc/dec engaged. [ 3.340868] AES CTR mode by8 optimization enabled [ 3.417649] sched_clock: Marking stable (3417566207, 0)->(4262100340, -844534133) [ 3.420154] registered taskstats version 1 [ 3.421462] Loading compiled-in X.509 certificates [ 3.422619] zswap: loaded using pool lzo/zbud [ 3.439379] Key type big_key registered [ 3.448738] Key type encrypted registered [ 3.450172] ima: No TPM chip found, activating TPM-bypass! [ 3.451312] ima: Allocated hash algorithm: sha1 [ 3.452300] ima: No architecture policies found [ 3.453320] evm: Initialising EVM extended attributes: [ 3.454350] evm: security.selinux [ 3.455038] evm: security.ima [ 3.455559] evm: security.capability [ 3.456342] evm: HMAC attrs: 0x1 [ 3.457830] rtc_cmos 00:05: setting system clock to 2026-08-06 16:46:40 UTC (1786034800) [ 3.461729] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 3.463596] debug: unmapping init [mem 0xffffffffa6782000-0xffffffffa6a58fff] [ 3.471115] Write protecting the kernel read-only data: 28672k [ 3.473563] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 3.475493] debug: unmapping init [mem 0xffffffffa5714000-0xffffffffa57fffff] [ 3.513299] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.523315] systemd[1]: Detected virtualization kvm. [ 3.525282] systemd[1]: Detected architecture x86-64. [ 3.527251] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.551134] systemd[1]: No hostname configured. [ 3.553134] systemd[1]: Set hostname to . [ 3.554592] random: systemd: uninitialized urandom read (16 bytes read) [ 3.557379] systemd[1]: Initializing machine ID from random generator. [ 3.707543] random: systemd: uninitialized urandom read (16 bytes read) [ 3.710314] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.714162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.717048] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.722216] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.411872] device-mapper: uevent: version 1.0.3 [ 4.414135] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.229487] virtio_net virtio0 ens2: renamed from eth0 [ 5.258097] random: fast init done [ 5.295544] scsi host0: ata_piix [ 5.333962] scsi host1: ata_piix [ 5.335901] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.338557] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.975604] random: crng init done [ 9.978167] random: 7 urandom warning(s) missed due to ratelimiting [ 10.312455] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.771213] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.918860] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.002432] SELinux: Disabled at runtime. [ 16.095237] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.111841] systemd[1]: Detected virtualization kvm. [ 16.114467] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.222353] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.231479] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.243388] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.252081] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.260836] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.286469] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.341885] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ 18.534811] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 20.250203] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.526057] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.614295] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.263528] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.412440] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 27.801459] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 28.633203] NFS: Registering the id_resolver key type [ 28.638278] Key type id_resolver registered [ 28.639639] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg130-server login: [ 42.287013] hrtimer: interrupt took 10282373 ns [ 86.389930] libcfs: loading out-of-tree module taints kernel. [ 86.432843] Key type ._llcrypt registered [ 86.435130] Key type .llcrypt registered [ 86.544279] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 106.370880] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 108.313456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.349621] alg: No test for adler32 (adler32-zlib) [ 110.097245] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 111.343750] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 113.272403] Key type lgssc registered [ 115.382921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.184609] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.848962] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 200.275366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.384710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.728200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.779546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.890928] Lustre: lustre-MDT0000: new disk, initializing [ 201.980310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.000462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.343692] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.100197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.247825] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.347424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.356586] Lustre: Skipped 1 previous similar message [ 219.479822] Lustre: lustre-MDT0001: new disk, initializing [ 219.567119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.620893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.642449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.380473] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.364862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.892158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.455330] Lustre: lustre-OST0000: new disk, initializing [ 242.464947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.472855] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.643717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.769515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.799686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.972613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.385487] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.623955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.891298] Lustre: lustre-OST0001: new disk, initializing [ 269.903415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.917948] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.064426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.070327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.086652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.207181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.967293] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.175749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.681207] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.594993] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.669424] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing yml_node [ 315.782465] Lustre: DEBUG MARKER: Client: 2.17.56.52 [ 318.893749] Lustre: DEBUG MARKER: MDS: 2.17.56.52 [ 321.484348] Lustre: DEBUG MARKER: OSS: 2.17.56.52 [ 323.489790] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 12:51:59 EDT 2026 [ 342.069496] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.713685] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.394525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.395609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.411993] Lustre: Skipped 3 previous similar messages [ 359.419433] Lustre: Skipped 2 previous similar messages [ 364.517582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.525678] Lustre: Skipped 4 previous similar messages [ 365.801810] Lustre: server umount lustre-MDT0000 complete [ 369.870858] LustreError: 9522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035166 with bad export cookie 3554181795402736397 [ 369.889904] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.890419] LustreError: 9522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 374.753194] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.753770] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.754486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.767952] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 374.811635] Lustre: Skipped 1 previous similar message [ 376.412510] Lustre: server umount lustre-MDT0001 complete [ 380.459509] Lustre: server umount lustre-OST0000 complete [ 384.383079] Lustre: server umount lustre-OST0001 complete [ 392.741822] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 402.843062] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 443.511819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.973384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.068674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.222378] Lustre: lustre-MDT0000: new disk, initializing [ 444.390320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.406876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.462614] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 463.233103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.384453] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 463.466887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 463.471703] Lustre: Skipped 1 previous similar message [ 463.624168] Lustre: lustre-MDT0001: new disk, initializing [ 463.821947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.909068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.942858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.969548] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 478.971016] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.204820] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.443890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.717948] Lustre: lustre-OST0000: new disk, initializing [ 491.720787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.725224] Lustre: 17054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.795835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.606880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 493.620645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 493.795234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 498.190754] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 507.757808] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.638190] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.907170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.967717] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.265708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 524.770866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.780407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.780653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.783237] Lustre: Skipped 1 previous similar message [ 526.691663] Lustre: server umount lustre-OST0000 complete [ 530.401601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.413052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.431919] Lustre: Skipped 1 previous similar message [ 530.450608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.453517] Lustre: Skipped 1 previous similar message [ 535.009580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.601982] Lustre: server umount lustre-MDT0000 complete [ 540.150552] LustreError: 16368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.195368] LustreError: 16368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 540.984665] LustreError: 15879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035338 with bad export cookie 3554181795402737930 [ 540.987704] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.994058] LustreError: 15879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 541.244123] Lustre: server umount lustre-MDT0001 complete [ 545.382320] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:55:40 (1786035340) [ 548.917667] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 552.363538] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:55:47 (1786035347) [ 561.235667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.911396] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.029646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.039697] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 567.267274] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.290143] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 571.362939] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.260464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.601310] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 592.971202] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.936161] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 608.288631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.945391] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 624.016489] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 646.626819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.638331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.645257] Lustre: Skipped 1 previous similar message [ 646.657813] Lustre: Skipped 3 previous similar messages [ 649.869694] Lustre: server umount lustre-OST0000 complete [ 654.818523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 659.563585] Lustre: server umount lustre-MDT0000 complete [ 662.007446] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.058354] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 664.147251] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035461 with bad export cookie 3554181795402738812 [ 664.171986] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.335408] Lustre: server umount lustre-MDT0001 complete [ 679.807975] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 682.869882] Key type lgssc unregistered [ 683.205892] LNet: 23023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.221746] LNetError: 23023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.255826] LNet: Removed LNI 192.168.201.130@tcp [ 684.301176] Key type .llcrypt unregistered [ 684.307624] Key type ._llcrypt unregistered [ 711.082430] Key type ._llcrypt registered [ 711.087124] Key type .llcrypt registered [ 711.186458] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 712.459837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.498393] alg: No test for adler32 (adler32-zlib) [ 713.666957] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 713.977831] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 715.691593] Key type lgssc registered [ 716.603242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.213560] loop: module loaded [ 726.271389] loop0: detected capacity change from 0 to 204800000 [ 726.490173] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.504851] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.733423] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.490521] loop0: detected capacity change from 0 to 204800000 [ 733.778576] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.802624] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.814231] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.494835] loop0: detected capacity change from 0 to 204800000 [ 744.957790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 744.983689] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.230037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 746.345918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 746.471210] Lustre: lustre-MDT0000: new disk, initializing [ 746.634639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.683041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 751.174782] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 760.400319] loop1: detected capacity change from 0 to 204800000 [ 766.626685] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 766.999577] Lustre: lustre-OST07c6: new disk, initializing [ 767.001980] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 767.008575] Lustre: Skipped 1 previous similar message [ 767.020353] Lustre: 26209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 767.128788] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 774.847528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 774.868946] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 774.952287] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 774.955195] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 780.345785] Lustre: 25198:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 780.363727] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 780.394973] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 780.415499] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 780.424918] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 780.442183] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 794.700200] Lustre: 26834:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 794.707920] Lustre: 26834:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.715417] Lustre: 26834:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.723611] Lustre: 26834:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 794.735376] Lustre: 26834:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 794.745052] Lustre: 26834:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 800.016747] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 800.025743] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 800.032438] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 800.038155] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.044788] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 800.049840] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.054387] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 800.061358] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.067946] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 800.079560] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.087157] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 800.094506] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.313751] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 803.336562] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 803.346947] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 803.351663] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.369035] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.388246] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.395662] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 803.404458] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.413250] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 803.418296] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.423631] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.428769] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.720821] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 811.726050] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 811.731398] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.740025] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.744089] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.749360] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.753662] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.758485] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.763000] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 811.767366] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.771186] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.775178] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 823.021429] Lustre: 26834:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 823.036186] Lustre: 26834:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 823.046750] Lustre: 26834:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 823.053305] Lustre: 26834:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.062140] Lustre: 26834:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.068533] Lustre: 26834:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.087243] Lustre: 26834:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 823.096751] Lustre: 26834:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.102368] Lustre: 26834:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 823.122333] Lustre: 26834:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.140439] Lustre: 26834:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 823.157719] Lustre: 26834:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 867.297736] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.308302] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.318415] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 871.905606] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 877.027983] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 881.122673] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 881.275620] Lustre: server umount lustre-OST07c6 complete [ 891.237873] Lustre: server umount lustre-MDT0000 complete [ 909.932469] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 912.473433] Key type lgssc unregistered [ 912.811640] LNet: 28781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.818653] LNetError: 28781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.836826] LNet: Removed LNI 192.168.201.130@tcp [ 913.611830] Key type .llcrypt unregistered [ 913.613840] Key type ._llcrypt unregistered [ 933.529618] Key type ._llcrypt registered [ 933.532279] Key type .llcrypt registered [ 933.661561] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 935.140220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.431635] alg: No test for adler32 (adler32-zlib) [ 936.627129] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 937.005187] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 938.704207] Key type lgssc registered [ 940.046512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.754749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.787954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.211107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.857860] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 959.457670] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.470367] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 964.324136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.719728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.723510] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 979.197229] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.448526] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.558545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.021420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.090703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 999.650736] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1010.739422] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.786601] Lustre: server umount lustre-OST0000 complete [ 1035.235041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.247823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.260274] Lustre: Skipped 1 previous similar message [ 1035.282975] Lustre: Skipped 1 previous similar message [ 1040.357170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.371575] Lustre: Skipped 1 previous similar message [ 1044.553663] Lustre: server umount lustre-MDT0000 complete [ 1045.478493] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.506109] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1046.508304] Lustre: 29361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786035827/real 1786035827] req@000000000a317cdb x1872794205847680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786035843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1046.508602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.541144] Lustre: 29361:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1046.597879] LustreError: 30050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.622901] LustreError: 30050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1048.944348] LustreError: 31898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035845 with bad export cookie 6214461560861657444 [ 1048.951363] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.953401] LustreError: 31898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1049.345938] Lustre: server umount lustre-MDT0001 complete [ 1062.322658] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1065.212690] Key type lgssc unregistered [ 1065.588909] LNet: 33789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.595139] LNetError: 33789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.671491] LNet: Removed LNI 192.168.201.130@tcp [ 1067.730217] Key type .llcrypt unregistered [ 1067.732084] Key type ._llcrypt unregistered [ 1087.609675] Key type ._llcrypt registered [ 1087.612672] Key type .llcrypt registered [ 1087.692489] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1088.523771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.699677] alg: No test for adler32 (adler32-zlib) [ 1090.070360] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1090.455503] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1092.280187] Key type lgssc registered [ 1093.601444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.633626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.651630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.191426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.329755] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1113.570758] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.599982] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1116.882160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.181508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.312075] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1130.248048] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.701986] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1146.589577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.989268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.034628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1153.836239] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1162.932366] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.093147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.099299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.100991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.125288] Lustre: Skipped 1 previous similar message [ 1180.691751] Lustre: server umount lustre-OST0000 complete [ 1188.322667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.327485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.332861] Lustre: Skipped 1 previous similar message [ 1188.337781] Lustre: Skipped 1 previous similar message [ 1193.442895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.454338] Lustre: Skipped 2 previous similar messages [ 1198.563579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.566659] Lustre: Skipped 1 previous similar message [ 1199.072139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1199.467514] Lustre: server umount lustre-MDT0000 complete [ 1203.681822] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.693484] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1204.176405] LustreError: 35056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036001 with bad export cookie 9219042160911473077 [ 1204.191234] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.192268] LustreError: 35056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1204.505470] Lustre: server umount lustre-MDT0001 complete [ 1219.363633] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1222.287336] Key type lgssc unregistered [ 1222.546817] LNet: 38616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.552694] LNetError: 38616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.565845] LNet: Removed LNI 192.168.201.130@tcp [ 1223.218717] Key type .llcrypt unregistered [ 1223.225131] Key type ._llcrypt unregistered [ 1246.869895] Key type ._llcrypt registered [ 1246.873475] Key type .llcrypt registered [ 1247.035892] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1248.019977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.143079] alg: No test for adler32 (adler32-zlib) [ 1249.305544] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1249.605959] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1251.256186] Key type lgssc registered [ 1252.317761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.032913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.062577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.565395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.651507] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1271.784529] LustreError: 40032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.813174] LustreError: 40032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1275.279267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.803967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.530801] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1288.696975] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.380463] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1301.382842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.618375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.734266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1307.418667] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1315.688504] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1325.538195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.555919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.564861] Lustre: Skipped 2 previous similar messages [ 1327.857782] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1330.659654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.673169] Lustre: Skipped 2 previous similar messages [ 1331.382151] Lustre: server umount lustre-MDT0000 complete [ 1332.967235] LustreError: 40029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.405427] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036132 with bad export cookie 16975758506509395257 [ 1335.406569] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.418050] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1335.778684] LustreError: 40028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.780866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.793244] Lustre: Skipped 2 previous similar messages [ 1335.796540] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1338.058989] LustreError: 40027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.905686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1340.921162] Lustre: Skipped 1 previous similar message [ 1341.994861] Lustre: server umount lustre-MDT0001 complete [ 1357.381741] Lustre: server umount lustre-OST0000 complete [ 1375.007295] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1377.973745] Key type lgssc unregistered [ 1378.316553] LNet: 43895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.322352] LNetError: 43895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.372173] LNet: Removed LNI 192.168.201.130@tcp [ 1380.363283] Key type .llcrypt unregistered [ 1380.366375] Key type ._llcrypt unregistered [ 1407.870377] Key type ._llcrypt registered [ 1407.883430] Key type .llcrypt registered [ 1408.021882] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1409.628848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.676477] alg: No test for adler32 (adler32-zlib) [ 1410.869372] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1411.111820] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1412.840299] Key type lgssc registered [ 1414.105269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.994130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1423.039291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.998060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.411308] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1434.093545] LustreError: 45314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.109769] LustreError: 45314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1439.222468] LustreError: 45315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.835895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.167832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.779844] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1452.064623] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1457.933443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1465.244280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.542657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.639233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1471.358915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1479.628342] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1486.580968] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1487.330633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.341879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.361589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.405170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.408711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.421688] Lustre: Skipped 1 previous similar message [ 1495.522219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.540242] Lustre: Skipped 2 previous similar messages [ 1500.128119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1500.335234] Lustre: server umount lustre-OST0000 complete [ 1521.121944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.126808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.140628] Lustre: Skipped 1 previous similar message [ 1521.152156] Lustre: Skipped 2 previous similar messages [ 1525.125541] Lustre: server umount lustre-MDT0000 complete [ 1525.223355] LustreError: 46045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.254648] LustreError: 46045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1529.014542] LustreError: 45295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036326 with bad export cookie 9145950593440639880 [ 1529.025887] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.029525] LustreError: 45295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1535.371948] Lustre: server umount lustre-MDT0001 complete [ 1549.350583] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1552.228703] Key type lgssc unregistered [ 1552.616950] LNet: 49026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.623070] LNetError: 49026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.638395] LNet: Removed LNI 192.168.201.130@tcp [ 1553.630549] Key type .llcrypt unregistered [ 1553.633877] Key type ._llcrypt unregistered [ 1581.517777] Key type ._llcrypt registered [ 1581.520326] Key type .llcrypt registered [ 1581.622493] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1582.690902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.741681] alg: No test for adler32 (adler32-zlib) [ 1583.912867] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1584.183522] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1585.824295] Key type lgssc registered [ 1586.828730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1596.164372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1596.192333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.929657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.333299] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1607.139472] LustreError: 50451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.178106] LustreError: 50451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1612.260895] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.354396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.979907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1618.522280] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1629.521428] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1636.369590] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1642.608601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.134601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.682850] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1658.577092] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1667.700714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1673.542156] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1677.542551] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1690.509713] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-0613h14m44s [ 1692.401543] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1696.737460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1696.744627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.765403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.838558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.857882] Lustre: Skipped 4 previous similar messages [ 1700.078164] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 1702.006139] Lustre: server umount lustre-MDT0000 complete [ 1703.910549] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.929025] LustreError: 50452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1705.840750] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1707.219548] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.070650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.292963] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.361758] LustreError: 53162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.400681] LustreError: 53162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1713.788449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.910610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1714.798537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1718.538721] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1722.969144] Lustre: DEBUG MARKER: Wait for df (36709) ... [ 1725.188552] Lustre: DEBUG MARKER: done [ 1732.577311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.588789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.599496] Lustre: Skipped 2 previous similar messages [ 1732.614876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.618942] Lustre: Skipped 1 previous similar message [ 1734.148199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.161331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.773974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.781982] Lustre: Skipped 4 previous similar messages [ 1745.376130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1745.724638] Lustre: server umount lustre-OST0000 complete [ 1751.014928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.015604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.026906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.049615] Lustre: Skipped 1 previous similar message [ 1751.066132] Lustre: Skipped 1 previous similar message [ 1757.215436] Lustre: server umount lustre-MDT0000 complete [ 1758.178437] LustreError: 53162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.194434] LustreError: 53162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1762.270778] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036559 with bad export cookie 10048070463821158314 [ 1762.271616] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1762.292418] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1762.697431] Lustre: server umount lustre-MDT0001 complete [ 1775.607435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1778.270558] Key type lgssc unregistered [ 1778.558778] LNet: 55403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.565121] LNetError: 55403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.639316] LNet: Removed LNI 192.168.201.130@tcp [ 1780.598794] Key type .llcrypt unregistered [ 1780.601100] Key type ._llcrypt unregistered [ 1825.030894] Key type ._llcrypt registered [ 1825.036173] Key type .llcrypt registered [ 1825.167455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1826.826689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.886818] alg: No test for adler32 (adler32-zlib) [ 1828.025879] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1828.331993] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1830.008336] Key type lgssc registered [ 1831.286697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1838.934318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1838.961071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.320620] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1840.343385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1840.529874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.218668] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1855.383990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.494819] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1855.515737] Lustre: 57612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1855.690304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.015190] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1868.996350] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.227548] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.092388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.401072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1883.652988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.173536] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1892.016032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1899.683208] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1908.293760] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-0613h18m23s [ 1910.427590] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1918.016981] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1920.747513] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1920.823903] LustreError: 57620:0:(mdt_handler.c:2379:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1941.231575] Lustre: lustre-MDT0000: Client 16680979-8fcc-46c1-ac37-49bd68346c0c (at 192.168.201.30@tcp) reconnecting [ 1960.858880] LustreError: 57620:0:(mdt_handler.c:2379:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1963.505350] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1966.166391] Lustre: DEBUG MARKER: done [ 1972.199464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.205767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.220425] Lustre: Skipped 1 previous similar message [ 1972.243721] Lustre: Skipped 1 previous similar message [ 1974.029757] Lustre: server umount lustre-OST0000 complete [ 1978.851529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.864866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.893370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.938720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.949316] Lustre: Skipped 2 previous similar messages [ 1983.970054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.978175] Lustre: Skipped 1 previous similar message [ 1990.115840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.119439] Lustre: Skipped 1 previous similar message [ 1992.160162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1994.377289] Lustre: server umount lustre-MDT0000 complete [ 1998.244477] LustreError: 58751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.276595] LustreError: 58751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1998.823878] LustreError: 58354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036795 with bad export cookie 6530479554358296539 [ 1998.828456] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.837529] LustreError: 58354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1999.392156] Lustre: server umount lustre-MDT0001 complete [ 2012.929988] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2016.178860] Key type lgssc unregistered [ 2016.544083] LNet: 61949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.565284] LNetError: 61949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.638954] LNet: Removed LNI 192.168.201.130@tcp [ 2018.516410] Key type .llcrypt unregistered [ 2018.521549] Key type ._llcrypt unregistered [ 2068.241196] Key type ._llcrypt registered [ 2068.245630] Key type .llcrypt registered [ 2068.325286] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2069.765275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.800547] alg: No test for adler32 (adler32-zlib) [ 2070.870644] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2071.098361] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2072.760185] Key type lgssc registered [ 2073.684460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2076.754804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.778811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.115937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2078.141178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2078.451341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.667284] Lustre: Failing over lustre-MDT0000 [ 2079.997585] Lustre: server umount lustre-MDT0000 complete [ 2087.963525] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:21:23 (1786036883) [ 2096.087595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.447901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.320587] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2111.984091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.056299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2112.085938] Lustre: 64748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2112.304220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.628612] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2125.582398] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.502980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2140.589401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.716853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2140.864912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.615029] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2150.561470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2156.902503] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.497531] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2173.924958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.938301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.963335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.443757] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.455948] Lustre: Skipped 1 previous similar message [ 2174.468851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.476070] Lustre: Skipped 2 previous similar messages [ 2177.939518] Lustre: server umount lustre-MDT0000 complete [ 2182.316520] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036979 with bad export cookie 3402153895274734121 [ 2182.325883] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.337105] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2182.889803] Lustre: server umount lustre-MDT0001 complete [ 2187.196798] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2191.949616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2200.035384] Lustre: 63672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786036981/real 1786036981] req@00000000d25b4901 x1872795394943104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786036997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.061572] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.168111] Lustre: 63672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786036984/real 1786036984] req@00000000da45385e x1872795394943232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786037000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.537900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.152356] Lustre: 63673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786036986/real 1786036986] req@000000000edd369d x1872795394943360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786037002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2207.740914] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.781461] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2207.880078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.021943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2208.800163] Lustre: 63674:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786036989/real 1786036989] req@000000006816e523 x1872795394943616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786037005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2209.897708] LustreError: 66597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.934819] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.962727] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2212.808676] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2213.985325] Lustre: 63671:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786036995/real 1786036995] req@00000000d892d13e x1872795394943744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786037011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2216.427066] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.448424] LustreError: 68516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2221.539457] LustreError: 68514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.555213] LustreError: 68514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2222.045307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.467388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.708222] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2227.795888] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2229.809858] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2236.076266] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.289544] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2250.729434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.734863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.741920] Lustre: Skipped 1 previous similar message [ 2250.750192] Lustre: Skipped 2 previous similar messages [ 2255.846203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.855081] Lustre: Skipped 2 previous similar messages [ 2256.973722] Lustre: server umount lustre-MDT0000 complete [ 2260.965672] LustreError: 66597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.990877] LustreError: 66597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2262.572961] LustreError: 64739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037059 with bad export cookie 3402153895274736396 [ 2262.583745] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.591272] LustreError: 64739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2263.433416] Lustre: server umount lustre-MDT0001 complete [ 2271.303725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2281.472615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.464227] Lustre: 63674:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786037063/real 1786037063] req@000000001a37fdbb x1872795394971520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786037079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2282.476367] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.482778] Lustre: Skipped 1 previous similar message [ 2287.589914] LustreError: 63670:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f3b0e96 x1872795394972544/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2287.954302] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.999582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.055957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2291.068631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2292.710922] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2301.384215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.129228] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2307.055707] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2314.734034] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2321.035432] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2332.655617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.657216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.658549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.666711] Lustre: Skipped 1 previous similar message [ 2334.778061] Lustre: server umount lustre-MDT0000 complete [ 2337.764060] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.779737] LustreError: 71200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2338.630639] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037135 with bad export cookie 3402153895274737775 [ 2338.636173] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.648735] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2339.127709] Lustre: server umount lustre-MDT0001 complete [ 2346.692851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2350.420408] Lustre: DEBUG MARKER: files compared the same [ 2363.377366] Lustre: server umount lustre-OST0000 complete [ 2382.139309] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2384.750979] Key type lgssc unregistered [ 2385.073818] LNet: 74478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.077431] LNetError: 74478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.090390] LNet: Removed LNI 192.168.201.130@tcp [ 2385.991097] Key type .llcrypt unregistered [ 2385.994246] Key type ._llcrypt unregistered [ 2413.582609] Key type ._llcrypt registered [ 2413.590337] Key type .llcrypt registered [ 2413.758627] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2415.323161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2415.360617] alg: No test for adler32 (adler32-zlib) [ 2416.576924] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2416.923743] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2418.706125] Key type lgssc registered [ 2419.862713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.143457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2428.158171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.590450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.860573] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2439.139803] LustreError: 75894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.171637] LustreError: 75894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2443.632549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.864448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2449.185348] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2458.013304] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2463.694849] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2470.503728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.900885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.992668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2479.070439] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug +malloc all [ 2487.783695] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2498.530305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.532887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.545297] Lustre: Skipped 1 previous similar message [ 2498.546022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.385884] Lustre: server umount lustre-OST0000 complete [ 2508.304181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.317648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.329649] Lustre: Skipped 2 previous similar messages [ 2513.386855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.276773] Lustre: server umount lustre-MDT0000 complete [ 2518.500906] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.841397] LustreError: 75877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037316 with bad export cookie 4581578868831137461 [ 2519.842227] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.850269] LustreError: 75877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2520.177806] Lustre: server umount lustre-MDT0001 complete [ 2532.889794] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2536.021860] Key type lgssc unregistered [ 2536.364690] LNet: 79431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.374210] LNetError: 79431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.446444] LNet: Removed LNI 192.168.201.130@tcp [ 2538.357247] Key type .llcrypt unregistered [ 2538.358979] Key type ._llcrypt unregistered [ 2565.278386] Key type ._llcrypt registered [ 2565.284498] Key type .llcrypt registered [ 2565.485238] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2567.811493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2568.432776] alg: No test for adler32 (adler32-zlib) [ 2570.153779] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2570.656446] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2572.344520] Key type lgssc registered [ 2574.748507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2585.629491] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2585.669404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.376933] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2606.496966] LustreError: 80825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [ 2633.184232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.249984] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2648.609660] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2656.441914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.619458] Lustre: *** cfs_fail_loc=706, val=0*** [ 2657.193226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.326213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2662.383050] LustreError: 80850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.409969] LustreError: 80850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2662.422367] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2667.494319] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.512693] LustreError: 81909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2672.498541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.614802] LustreError: 81910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.647891] LustreError: 81910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2673.038577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.964654] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2686.299855] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2692.644320] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2698.210721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2698.214806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.222425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2698.721617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2698.745103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.760064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2701.372132] Lustre: server umount lustre-OST0000 complete [ 2708.962821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.969726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.995864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.015285] Lustre: Skipped 1 previous similar message [ 2711.873938] Lustre: server umount lustre-MDT0000 complete [ 2714.085389] LustreError: 81914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.132902] LustreError: 81914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2717.939461] LustreError: 82652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037514 with bad export cookie 15268086121395747091 [ 2717.953449] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.964694] LustreError: 82652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2718.470400] Lustre: server umount lustre-MDT0001 complete [ 2733.708524] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 2736.995300] Key type lgssc unregistered [ 2737.345639] LNet: 84445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.355526] LNetError: 84445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2738.409701] LNet: Removed LNI 192.168.201.130@tcp [ 2739.804305] Key type .llcrypt unregistered [ 2739.807891] Key type ._llcrypt unregistered [ 2767.680899] Key type ._llcrypt registered [ 2767.684426] Key type .llcrypt registered [ 2767.825597] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2769.064591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2769.115266] alg: No test for adler32 (adler32-zlib) [ 2770.239684] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2770.525855] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 2772.160183] Key type lgssc registered [ 2773.752454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2781.271361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2781.309714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.976990] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2795.076465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.510684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.151736] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2800.627240] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.658281] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2805.730403] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.741702] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2809.188041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.396818] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.421858] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2809.445870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.949444] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.979427] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2815.709628] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2820.070728] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.080306] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2824.093612] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.109275] Lustre: MGS: Client 186ffcc1-b6a4-43a2-aff2-55bd2f12c547 (at 0@lo) reconnecting [ 2824.145467] LustreError: 86563:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2824.390149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.489957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2828.896902] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2845.170558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2845.174105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.198186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.227815] Lustre: Skipped 1 previous similar message [ 2845.247899] Lustre: Skipped 1 previous similar message [ 2849.196863] Lustre: server umount lustre-OST0000 complete [ 2855.393750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2855.397479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.403693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2855.424591] Lustre: Skipped 1 previous similar message [ 2859.682870] Lustre: server umount lustre-MDT0000 complete [ 2860.516754] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.541343] LustreError: 86549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2864.043150] Lustre: server umount lustre-MDT0001 complete [ 2867.829860] Lustre: server umount MGS complete [ 2880.991802] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:34:36 (1786037676) [ 2916.071529] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 2930.435205] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2983.956095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.942233] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2998.876566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.209761] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2999.281554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2999.392762] Lustre: lustre-MDT0001: new disk, initializing [ 2999.510047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2999.542484] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2999.555369] Lustre: 93720:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2999.570016] LustreError: 93720:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3004.816116] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3017.949089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3018.228891] Lustre: lustre-OST0000: new disk, initializing [ 3018.232475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3018.237461] Lustre: 94485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3018.292486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.028169] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3029.729965] Lustre: 94927:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000003faaf83 x1872796127894400/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786037842 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3038.175816] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.742837] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.753301] Lustre: MGS: Client 9a467c7b-3cef-4ddc-8257-9deb55945acd (at 0@lo) reconnecting [ 3045.766880] LustreError: 93706:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3045.800959] Lustre: 95531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3045.836657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3045.915377] Lustre: lustre-MDT0000: new disk, initializing [ 3046.105135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.147237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3046.200339] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3047.296759] Lustre: 95555:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d2601709 x1872796127901568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786037860 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3051.844234] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3052.470858] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3056.450279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3056.470260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3056.627525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3080.165665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.167064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3080.185419] Lustre: Skipped 1 previous similar message [ 3080.199399] Lustre: Skipped 2 previous similar messages [ 3085.262937] Lustre: server umount lustre-OST0000 complete [ 3095.532695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.535840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3095.543632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.549781] Lustre: Skipped 1 previous similar message [ 3097.849803] Lustre: server umount lustre-MDT0000 complete [ 3100.645816] LustreError: 95899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.659503] LustreError: 95899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3108.467875] Lustre: server umount lustre-MDT0001 complete [ 3112.379696] Lustre: server umount MGS complete [ 3124.403955] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:38:40 (1786037920) [ 3144.796763] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3147.616889] Key type lgssc unregistered [ 3147.871750] LNet: 98059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3147.879968] LNetError: 98059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3147.891697] LNet: Removed LNI 192.168.201.130@tcp [ 3148.887876] Key type .llcrypt unregistered [ 3148.889270] Key type ._llcrypt unregistered [ 3168.215856] Key type ._llcrypt registered [ 3168.219036] Key type .llcrypt registered [ 3168.326898] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3169.761803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3170.049111] alg: No test for adler32 (adler32-zlib) [ 3171.046920] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3171.254727] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3172.960829] Key type lgssc registered [ 3174.179774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3182.015347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3182.037424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.478080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.545474] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3193.834804] LustreError: 99280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.848061] LustreError: 99280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3197.163735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.509489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.906995] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3211.458398] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3218.612753] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3226.888652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.429098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.499871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3237.241608] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3250.022633] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3264.483028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3264.485485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.493497] LustreError: Skipped 1 previous similar message [ 3264.509234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.518053] Lustre: Skipped 1 previous similar message [ 3264.553845] Lustre: Skipped 1 previous similar message [ 3269.602820] LustreError: 101615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.748896] Lustre: server umount lustre-OST0000 complete [ 3274.216109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3274.228838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.243100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.846983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.858750] Lustre: Skipped 2 previous similar messages [ 3280.161465] Lustre: server umount lustre-MDT0000 complete [ 3284.899624] LustreError: 99262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038081 with bad export cookie 3137400508500035076 [ 3284.919111] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.402391] Lustre: server umount lustre-MDT0001 complete [ 3300.731378] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3304.047285] Key type lgssc unregistered [ 3304.540682] LNet: 102841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3304.559015] LNetError: 102841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.577478] LNet: Removed LNI 192.168.201.130@tcp [ 3306.068375] Key type .llcrypt unregistered [ 3306.071699] Key type ._llcrypt unregistered [ 3326.307677] Key type ._llcrypt registered [ 3326.311514] Key type .llcrypt registered [ 3326.487823] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3328.241222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.326410] alg: No test for adler32 (adler32-zlib) [ 3329.619660] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3329.993740] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3331.888488] Key type lgssc registered [ 3334.122381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3342.567684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3342.574187] LustreError: 103935:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3342.592564] LustreError: 103938:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3342.599547] LustreError: 103935:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3342.602925] LustreError: 103935:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3342.608523] LustreError: 103935:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3342.649230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.466792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.479895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.601358] LustreError: 103976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.626901] LustreError: 103976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3354.951344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.568585] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3367.644388] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing lsmod [ 3371.139673] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3371.153383] LustreError: 105145:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3371.157868] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3371.165394] LustreError: 105144:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3371.172667] LustreError: 105144:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3371.175100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.635330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.586067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3375.601699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.652412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.122034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.131470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.236380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.248432] Lustre: Skipped 4 previous similar messages [ 3381.653547] Lustre: server umount lustre-MDT0000 complete [ 3384.289662] LustreError: 103971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.315365] LustreError: 103971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3385.698049] LustreError: 103957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038182 with bad export cookie 16310843286674986896 [ 3385.720534] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3387.488465] LustreError: 104407:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007f3b0e96 x1872796715078400/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3387.792709] Lustre: server umount lustre-MDT0001 complete [ 3402.522667] Lustre: server umount lustre-OST0000 complete [ 3412.074945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.699382] LustreError: 106209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.775839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.081909] LustreError: 106210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.906524] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3422.195170] LustreError: 106209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.427549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.905462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3433.779513] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3443.963247] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3452.347209] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3461.756705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.402528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.742737] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3479.606339] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3482.879958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3493.346551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3493.349882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.351993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.356582] Lustre: Skipped 2 previous similar messages [ 3493.365362] Lustre: Skipped 1 previous similar message [ 3498.488472] Lustre: server umount lustre-OST0000 complete [ 3503.587868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.604548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.612229] Lustre: Skipped 1 previous similar message [ 3513.826025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.835660] Lustre: Skipped 2 previous similar messages [ 3516.384233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3516.522178] Lustre: server umount lustre-MDT0000 complete [ 3518.947168] LustreError: 106210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.964706] LustreError: 106210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3521.221549] LustreError: 106189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038318 with bad export cookie 16310843286674987596 [ 3521.229066] LustreError: 106189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3521.245167] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3521.970275] Lustre: server umount lustre-MDT0001 complete [ 3535.602791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3538.056545] Key type lgssc unregistered [ 3538.349823] LNet: 109770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.357533] LNetError: 109770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.371836] LNet: Removed LNI 192.168.201.130@tcp [ 3539.162476] Key type .llcrypt unregistered [ 3539.164198] Key type ._llcrypt unregistered [ 3570.619644] Key type ._llcrypt registered [ 3570.623454] Key type .llcrypt registered [ 3570.735496] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3571.958945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3572.044506] alg: No test for adler32 (adler32-zlib) [ 3573.400720] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3573.827071] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3575.600271] Key type lgssc registered [ 3577.287091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.311505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.332597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.756596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.690453] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3597.305217] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.327061] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3602.408247] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.344691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.726666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.780934] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3618.236591] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3625.960443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3635.078399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.694510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.643725] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3651.689636] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3657.277891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3664.594426] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3669.196712] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3674.595646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.610565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.636777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.670950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.674677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.686316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.907490] Lustre: server umount lustre-OST0000 complete [ 3686.883167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.890609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.909567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.914938] Lustre: Skipped 1 previous similar message [ 3692.605253] Lustre: server umount lustre-MDT0000 complete [ 3693.031967] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.052052] LustreError: 111190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3697.180195] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038494 with bad export cookie 72262485026924035 [ 3697.187811] LustreError: 111171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3697.189192] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.460674] Lustre: server umount lustre-MDT0001 complete [ 3712.372408] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3715.874469] Key type lgssc unregistered [ 3716.188036] LNet: 114856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3716.196081] LNetError: 114856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3716.212913] LNet: Removed LNI 192.168.201.130@tcp [ 3717.463683] Key type .llcrypt unregistered [ 3717.465485] Key type ._llcrypt unregistered [ 3742.332520] Key type ._llcrypt registered [ 3742.337975] Key type .llcrypt registered [ 3742.512691] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3744.949372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3744.991086] alg: No test for adler32 (adler32-zlib) [ 3746.196894] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3746.659235] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3748.472199] Key type lgssc registered [ 3750.941274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3763.452262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3763.497526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.335943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.990942] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3775.460820] LustreError: 116096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.490162] LustreError: 116096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3780.591373] LustreError: 116097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.278151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.880892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3791.487662] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3802.100764] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3810.115861] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3818.712921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.217250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.618456] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3835.698337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3836.106031] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3846.130700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.132537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.146436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.151671] Lustre: Skipped 1 previous similar message [ 3846.160797] Lustre: Skipped 1 previous similar message [ 3851.237498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.662803] Lustre: server umount lustre-OST0000 complete [ 3856.356836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.357876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3856.378344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.378351] Lustre: Skipped 1 previous similar message [ 3861.475164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.482479] Lustre: Skipped 2 previous similar messages [ 3866.593771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.600615] Lustre: Skipped 1 previous similar message [ 3870.177125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3870.515332] Lustre: server umount lustre-MDT0000 complete [ 3871.722203] LustreError: 117024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.739459] LustreError: 117024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3875.407121] LustreError: 116075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038672 with bad export cookie 10797153344499682933 [ 3875.416691] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.422031] LustreError: 116075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3875.949506] Lustre: server umount lustre-MDT0001 complete [ 3890.599967] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3893.680525] Key type lgssc unregistered [ 3894.016123] LNet: 119641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.022209] LNetError: 119641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3894.038348] LNet: Removed LNI 192.168.201.130@tcp [ 3894.913301] Key type .llcrypt unregistered [ 3894.915058] Key type ._llcrypt unregistered [ 3923.637689] Key type ._llcrypt registered [ 3923.640994] Key type .llcrypt registered [ 3923.749332] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3925.490296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3925.519864] alg: No test for adler32 (adler32-zlib) [ 3926.900227] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3927.207212] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3928.891116] Key type lgssc registered [ 3930.629399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3938.983725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3939.009994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.517775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.262542] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3950.059092] LustreError: 121061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.079990] LustreError: 121061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3955.174796] LustreError: 121062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.860517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.646104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3963.231812] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3972.087286] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3979.467868] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3986.961464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.450696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.977055] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4003.926156] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4011.010376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4018.670630] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4024.300569] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4024.313356] Lustre: Skipped 2 previous similar messages [ 4027.656029] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4027.723415] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4029.271414] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4029.274663] Lustre: Skipped 2 previous similar messages [ 4034.175485] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4034.180736] Lustre: Skipped 1 previous similar message [ 4039.944397] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4039.963424] Lustre: Skipped 2 previous similar messages [ 4041.989198] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.30@tcp 192.168.0.[2,10]@tcp [ 4051.992324] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.30/24@tcp in log lustre-MDT0000 [ 4051.997586] Lustre: Skipped 2 previous similar messages [ 4060.749494] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 4060.757090] Lustre: Skipped 1 previous similar message [ 4068.852248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4068.856062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.864554] LustreError: Skipped 1 previous similar message [ 4068.901137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.350513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.357733] LustreError: Skipped 1 previous similar message [ 4069.361506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.423346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.437878] Lustre: Skipped 2 previous similar messages [ 4075.025792] Lustre: server umount lustre-OST0000 complete [ 4080.609826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4080.624398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.641025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.658652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.666529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.699364] Lustre: Skipped 1 previous similar message [ 4086.400087] Lustre: server umount lustre-MDT0000 complete [ 4087.807375] LustreError: 122024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.821351] LustreError: 122024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4091.908527] LustreError: 121043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038888 with bad export cookie 13456252119390506901 [ 4091.918256] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.922267] LustreError: 121043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4092.453130] Lustre: server umount lustre-MDT0001 complete [ 4105.411826] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4109.197528] Key type lgssc unregistered [ 4109.674139] LNet: 125986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4109.680478] LNetError: 125986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4109.719499] LNet: Removed LNI 192.168.201.130@tcp [ 4110.595438] Key type .llcrypt unregistered [ 4110.597635] Key type ._llcrypt unregistered [ 4147.621768] Key type ._llcrypt registered [ 4147.624352] Key type .llcrypt registered [ 4147.829157] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4149.610131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.671532] alg: No test for adler32 (adler32-zlib) [ 4150.978364] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4151.302641] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4153.064634] Key type lgssc registered [ 4154.596201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4163.472461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4163.487744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.981323] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4165.017120] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.30/24@tcp [ 4165.030508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.917720] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4175.329332] LustreError: 127549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.349923] LustreError: 127549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4179.355654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.754514] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4179.792138] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.30/24@tcp [ 4179.843460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4184.822937] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4193.177844] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4200.161723] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4208.057269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4208.322507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.374510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4214.853756] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4223.214652] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4235.244260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.247515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.259693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.263172] Lustre: Skipped 1 previous similar message [ 4240.369179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4240.385152] Lustre: Skipped 2 previous similar messages [ 4241.787458] Lustre: server umount lustre-OST0000 complete [ 4250.596261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.614521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.616204] Lustre: Skipped 1 previous similar message [ 4250.627329] Lustre: Skipped 1 previous similar message [ 4255.715282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.724474] Lustre: Skipped 1 previous similar message [ 4260.833810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4260.840792] Lustre: Skipped 1 previous similar message [ 4261.344396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4263.588706] Lustre: server umount lustre-MDT0000 complete [ 4265.953607] LustreError: 127542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.983072] LustreError: 127542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4267.976128] LustreError: 127530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039065 with bad export cookie 14790078882720985691 [ 4267.985806] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.990297] LustreError: 127530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4268.554096] Lustre: server umount lustre-MDT0001 complete [ 4285.374807] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4289.133727] Key type lgssc unregistered [ 4289.713143] LNet: 131141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4289.736176] LNetError: 131141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4289.767490] LNet: Removed LNI 192.168.201.130@tcp [ 4290.952653] Key type .llcrypt unregistered [ 4290.954353] Key type ._llcrypt unregistered [ 4324.006495] Key type ._llcrypt registered [ 4324.011364] Key type .llcrypt registered [ 4324.216956] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4342.137421] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4343.224798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.522441] alg: No test for adler32 (adler32-zlib) [ 4344.769420] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4345.170088] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4346.960138] Key type lgssc registered [ 4348.251706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4387.493165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4387.541549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.788030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4388.830748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4388.939905] Lustre: lustre-MDT0000: new disk, initializing [ 4389.066926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.079990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4394.139721] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4407.711928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.892191] Lustre: 134544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4407.995438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4408.000217] Lustre: Skipped 1 previous similar message [ 4408.147220] Lustre: lustre-MDT0001: new disk, initializing [ 4408.403280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4408.466222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4408.498808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4414.316173] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4425.642247] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4433.507982] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4441.161661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.441843] Lustre: lustre-OST0000: new disk, initializing [ 4441.446899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4441.453832] Lustre: 136513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4441.556201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.133664] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4448.358820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4448.366647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4448.518100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4459.345794] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4463.420483] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4463.638813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4468.056671] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4468.269204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4474.337700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4474.339434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.355365] LustreError: Skipped 1 previous similar message [ 4474.371162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4474.382267] Lustre: Skipped 1 previous similar message [ 4476.295036] Lustre: server umount lustre-OST0000 complete [ 4484.577831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.589379] Lustre: Skipped 1 previous similar message [ 4484.594656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.599600] Lustre: Skipped 1 previous similar message [ 4486.523136] Lustre: server umount lustre-MDT0000 complete [ 4490.626202] LustreError: 134536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039287 with bad export cookie 17394464151209588379 [ 4490.652535] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.654603] LustreError: 134536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4491.065423] Lustre: server umount lustre-MDT0001 complete [ 4500.959268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.438970] LustreError: 138742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.462643] LustreError: 138742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4501.544059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.594734] LustreError: 138743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.612654] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4511.714153] LustreError: 138742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.862060] LustreError: 138743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.122462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.720856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4524.340688] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4533.639537] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4540.449849] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4547.314481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.652569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.187514] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4562.342691] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.690912] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4572.222560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4577.426494] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4577.656483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.580487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4586.840416] Lustre: lustre-OST0001: new disk, initializing [ 4586.846831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4586.850979] Lustre: 142143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4586.963607] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4593.258857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4593.266838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4593.382988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4593.627668] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4602.228414] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4602.430065] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4607.380468] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4607.625399] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4614.316213] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc9529eb800.ost_server_uuid 50 [ 4616.273062] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc9529eb800.ost_server_uuid in IDLE state after 0 sec [ 4627.797779] Lustre: server umount lustre-OST0001 complete [ 4629.474844] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4629.478478] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.491299] LustreError: Skipped 1 previous similar message [ 4629.523701] Lustre: Skipped 2 previous similar messages [ 4629.536785] LustreError: 140567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4629.557027] LustreError: 140567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4632.545406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.557452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.569719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4634.596089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4634.597050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4637.686587] Lustre: server umount lustre-OST0000 complete [ 4641.762347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4641.776847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.796566] Lustre: Skipped 1 previous similar message [ 4641.804623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.812685] Lustre: Skipped 1 previous similar message [ 4649.958499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.974787] Lustre: Skipped 3 previous similar messages [ 4656.096315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4656.343890] Lustre: server umount lustre-MDT0000 complete [ 4659.176938] LustreError: 140178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.213813] LustreError: 140178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4660.683093] LustreError: 138725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039457 with bad export cookie 17394464151209589268 [ 4660.695049] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4660.707885] LustreError: 138725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4661.114453] Lustre: server umount lustre-MDT0001 complete [ 4684.622815] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4687.811877] Key type lgssc unregistered [ 4688.216288] LNet: 145153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4688.229149] LNetError: 145153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4688.265523] LNet: Removed LNI 192.168.201.130@tcp [ 4689.201880] Key type .llcrypt unregistered [ 4689.204303] Key type ._llcrypt unregistered [ 4735.235254] Key type ._llcrypt registered [ 4735.237335] Key type .llcrypt registered [ 4735.372813] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4750.221396] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4751.224423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4751.276506] alg: No test for adler32 (adler32-zlib) [ 4752.327718] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4752.508743] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4754.152180] Key type lgssc registered [ 4755.366855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4789.460633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4789.491142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.751325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.801292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4790.918570] Lustre: lustre-MDT0000: new disk, initializing [ 4791.072548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.087640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4795.157845] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4806.048651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.230441] Lustre: 149309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4806.304808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4806.309207] Lustre: Skipped 1 previous similar message [ 4806.399672] Lustre: lustre-MDT0001: new disk, initializing [ 4806.571374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4806.664340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4806.677381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4811.315826] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4823.005912] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4829.661985] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4836.703047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4836.973542] Lustre: lustre-OST0000: new disk, initializing [ 4836.979197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4836.986456] Lustre: 151279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4837.083645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.140151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4843.154213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4843.238345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4845.704577] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4857.056680] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4862.410771] Lustre: Failing over lustre-OST0000 [ 4862.524856] Lustre: server umount lustre-OST0000 complete [ 4880.288135] Lustre: 147535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039661/real 1786039661] req@000000004bf2c96b x1872798206168576/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786039677 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4880.321472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.041358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.444897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.481635] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4881.534026] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4885.473507] Lustre: 147532:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039666/real 1786039666] req@000000001f1edd1e x1872798206169984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786039682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4885.499446] Lustre: 147532:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4888.670662] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4890.691703] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4890.693271] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4891.748309] Lustre: 147534:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039671/real 1786039671] req@00000000fd80fa6d x1872798206171264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786039687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4893.136047] Lustre: Failing over lustre-MDT0000 [ 4893.395748] Lustre: server umount lustre-MDT0000 complete [ 4893.665153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.671170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.683888] Lustre: Skipped 1 previous similar message [ 4895.207363] Lustre: 147534:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039676/real 1786039676] req@000000000e195eda x1872798206172288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786039692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4895.231630] Lustre: 147534:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4895.714596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.727029] Lustre: Skipped 1 previous similar message [ 4900.834846] LustreError: 149316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.857347] LustreError: 149316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4902.099855] LustreError: 149318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.093349] LustreError: 149317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.276433] LustreError: 149317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.288279] LustreError: 149317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4912.099119] Lustre: 147532:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039692/real 1786039692] req@000000008431317c x1872798206182016/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786039708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4912.162762] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.641227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.460616] LustreError: 149318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.480831] LustreError: 149318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4922.580054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.632480] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4923.596864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4927.414797] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4927.988827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4928.017325] Lustre: Skipped 1 previous similar message [ 4928.049992] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4928.108578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4937.185301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4937.199628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.209897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4938.217314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4938.226679] Lustre: Skipped 1 previous similar message [ 4942.540547] Lustre: server umount lustre-OST0000 complete [ 4948.466350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.467642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4948.468208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.476639] Lustre: Skipped 2 previous similar messages [ 4952.483054] Lustre: server umount lustre-MDT0000 complete [ 4953.568970] LustreError: 149321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.590366] LustreError: 149321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4956.242043] LustreError: 149302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039753 with bad export cookie 5072500359518550604 [ 4956.247293] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.251136] LustreError: 149302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4956.711824] Lustre: server umount lustre-MDT0001 complete [ 4968.470852] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4971.601483] Key type lgssc unregistered [ 4972.011745] LNet: 154944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4972.023301] LNetError: 154944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4973.097728] LNet: Removed LNI 192.168.201.130@tcp [ 4974.044958] Key type .llcrypt unregistered [ 4974.047485] Key type ._llcrypt unregistered [ 5014.839808] Key type ._llcrypt registered [ 5014.842669] Key type .llcrypt registered [ 5015.034828] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5016.958688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5017.018222] alg: No test for adler32 (adler32-zlib) [ 5018.024747] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 5018.274426] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5019.968201] Key type lgssc registered [ 5021.125336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5030.025724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5030.037742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.499914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.467318] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5041.648608] LustreError: 156470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.685514] LustreError: 156470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5046.760531] LustreError: 156471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.724427] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5051.861295] LustreError: 156499:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5051.875487] LustreError: 156499:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5051.887962] LustreError: 156499:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 5052.244961] Lustre: server umount lustre-MDT0000 complete [ 5069.995102] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5081.015073] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5122.421979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.735790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5122.798426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5122.926142] Lustre: lustre-MDT0000: new disk, initializing [ 5123.069305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.094042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5128.168282] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5139.465092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.667449] Lustre: 159805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5139.778052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5139.804410] Lustre: Skipped 1 previous similar message [ 5140.008840] Lustre: lustre-MDT0001: new disk, initializing [ 5140.230981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.296348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5140.319126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5145.660430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5154.678176] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5161.757156] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.040184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.392489] Lustre: lustre-OST0000: new disk, initializing [ 5170.402463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5170.439174] Lustre: 161772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5170.556605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5177.018915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5177.413778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5177.419895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5177.539852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5187.617546] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5192.049172] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5194.338795] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:13:09 (1786039989) [ 5454.305603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.322757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.324302] Lustre: Skipped 1 previous similar message [ 5454.335053] Lustre: Skipped 2 previous similar messages [ 5459.425184] LustreError: 159811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.456410] LustreError: 159811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5459.608696] Lustre: server umount lustre-MDT0000 complete [ 5463.954667] LustreError: 159797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040260 with bad export cookie 2463480375466137201 [ 5463.962199] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.975106] LustreError: 159797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.544663] LustreError: 159812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.545250] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.604191] Lustre: Skipped 1 previous similar message [ 5464.614861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5468.577187] LustreError: 159813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.585878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5473.764854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5478.368389] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5478.955706] Lustre: server umount lustre-MDT0001 complete [ 5490.503943] Lustre: server umount lustre-OST0000 complete [ 5500.828648] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5513.021167] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5553.297439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.707612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5553.753153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5553.933709] Lustre: lustre-MDT0000: new disk, initializing [ 5554.051221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.075661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5558.374946] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5569.359581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.481526] Lustre: 166443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5569.532747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5569.536285] Lustre: Skipped 1 previous similar message [ 5569.608352] Lustre: lustre-MDT0001: new disk, initializing [ 5569.695783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.742892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5569.765523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5574.352472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5584.925622] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5591.975859] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5600.202904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5600.500919] Lustre: lustre-OST0000: new disk, initializing [ 5600.505109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5600.512739] Lustre: 168410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5600.609227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.392848] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5610.547619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5610.556099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5610.633917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5617.905964] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5622.657640] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5622.900556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5627.485575] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5627.659939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5630.955084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.964681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5630.974676] Lustre: Skipped 1 previous similar message [ 5630.987475] Lustre: Skipped 2 previous similar messages [ 5636.079647] Lustre: server umount lustre-OST0000 complete [ 5641.187264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.191728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5641.206781] Lustre: Skipped 1 previous similar message [ 5646.779922] Lustre: server umount lustre-MDT0000 complete [ 5650.803607] LustreError: 166436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040447 with bad export cookie 2463480375466296031 [ 5650.817051] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5651.149491] Lustre: server umount lustre-MDT0001 complete [ 5660.529746] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:20:56 (1786040456) [ 5676.914488] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5689.268669] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5729.406434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.662549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5729.701769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5729.794298] Lustre: lustre-MDT0000: new disk, initializing [ 5729.910671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.937780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5735.770474] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5745.387264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.577553] Lustre: 173172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5745.681804] LustreError: 173182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.710523] LustreError: 173182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5746.067351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5750.277986] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5759.070382] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.518713] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5773.501130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5773.686932] Lustre: lustre-OST0000: new disk, initializing [ 5773.689217] Lustre: Skipped 1 previous similar message [ 5773.692106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5773.694479] Lustre: Skipped 2 previous similar messages [ 5773.700121] Lustre: 175146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5780.977970] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5781.583446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5781.593597] Lustre: Skipped 1 previous similar message [ 5781.606712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5781.782908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5790.165359] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5797.358533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.367336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.372187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.372197] Lustre: Skipped 3 previous similar messages [ 5797.373674] Lustre: Skipped 1 previous similar message [ 5797.415779] LustreError: Skipped 1 previous similar message [ 5802.470246] LustreError: 175127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.483282] LustreError: 175127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5802.651389] Lustre: server umount lustre-OST0000 complete [ 5807.589861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.589964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5807.646579] Lustre: Skipped 1 previous similar message [ 5813.366594] Lustre: server umount lustre-MDT0000 complete [ 5817.632169] LustreError: 173163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040614 with bad export cookie 2463480375466296913 [ 5817.640704] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.641254] LustreError: 173163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5818.403894] Lustre: server umount lustre-MDT0001 complete [ 5827.233935] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:23:43 (1786040623) [ 5842.475235] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5853.604987] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5890.815178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.026564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5891.054828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5891.168907] Lustre: lustre-MDT0000: new disk, initializing [ 5891.240646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5891.249231] Lustre: Skipped 2 previous similar messages [ 5891.264618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5895.590135] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5906.906869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.121399] Lustre: 179487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5907.417158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5912.814988] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5922.815663] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5928.825946] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5935.872890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.286428] Lustre: 181465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5942.868407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5943.035045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5943.140429] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5953.391167] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5963.745615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.747248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.758862] LustreError: Skipped 1 previous similar message [ 5963.765031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.765041] Lustre: Skipped 5 previous similar messages [ 5963.799122] Lustre: Skipped 1 previous similar message [ 5968.106454] Lustre: server umount lustre-OST0000 complete [ 5973.986734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5986.272179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5986.612534] Lustre: server umount lustre-MDT0000 complete [ 5988.321497] LustreError: 180723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.338779] LustreError: 180723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5990.080645] LustreError: 179478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040787 with bad export cookie 2463480375466297998 [ 5990.084407] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.089984] LustreError: 179478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5990.434557] Lustre: server umount lustre-MDT0001 complete [ 6000.654850] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6003.666162] Key type lgssc unregistered [ 6003.941642] LNet: 183325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6003.950033] LNetError: 183325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6003.970800] LNet: Removed LNI 192.168.201.130@tcp [ 6004.837575] Key type .llcrypt unregistered [ 6004.839590] Key type ._llcrypt unregistered [ 6031.306415] Key type ._llcrypt registered [ 6031.308553] Key type .llcrypt registered [ 6031.439891] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6033.066575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6033.096600] alg: No test for adler32 (adler32-zlib) [ 6034.256728] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 6034.526793] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 6036.272221] Key type lgssc registered [ 6037.168367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6044.672555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6044.686341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.241620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.377277] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6056.429583] LustreError: 184740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.451951] LustreError: 184740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6061.245767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.575368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.295516] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6074.860676] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6081.591216] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6088.950499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6089.725616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.800129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6096.717680] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6104.434473] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6126.574857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.585809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6126.596728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.612585] Lustre: Skipped 1 previous similar message [ 6129.602690] Lustre: server umount lustre-OST0000 complete [ 6136.810627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.820857] Lustre: Skipped 2 previous similar messages [ 6136.827276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6136.837606] Lustre: Skipped 1 previous similar message [ 6140.169504] Lustre: server umount lustre-MDT0000 complete [ 6141.922430] LustreError: 184736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.951995] LustreError: 184736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6143.669923] LustreError: 184722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040940 with bad export cookie 9724961380095031453 [ 6143.684692] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.020485] Lustre: server umount lustre-MDT0001 complete [ 6155.761381] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6159.073729] Key type lgssc unregistered [ 6159.315947] LNet: 188301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.322178] LNetError: 188301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6159.339208] LNet: Removed LNI 192.168.201.130@tcp [ 6160.074934] Key type .llcrypt unregistered [ 6160.076713] Key type ._llcrypt unregistered [ 6189.502994] Key type ._llcrypt registered [ 6189.507838] Key type .llcrypt registered [ 6189.603628] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6190.579702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6190.606246] alg: No test for adler32 (adler32-zlib) [ 6191.828087] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 6192.234102] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 6193.936214] Key type lgssc registered [ 6195.011585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6203.057624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6203.082792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.769605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.431689] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6214.133299] LustreError: 189718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.152966] LustreError: 189718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6219.237633] LustreError: 189719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.407376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.790745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6225.419927] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6233.379427] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6238.556789] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6245.001041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.366660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6249.465764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6251.239301] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6259.198554] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6267.132709] Lustre: lustre-OST0000: Not available for connect from 192.168.201.30@tcp (stopping) [ 6269.922402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.930680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.932172] LustreError: Skipped 1 previous similar message [ 6269.936248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.973595] Lustre: Skipped 1 previous similar message [ 6272.195884] LustreError: 191543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6272.253959] Lustre: server umount lustre-OST0000 complete [ 6279.314807] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fc9606eb000.ost_server_uuid 50 [ 6280.912523] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6295.141242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.142222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.158165] Lustre: Skipped 1 previous similar message [ 6295.165922] Lustre: Skipped 1 previous similar message [ 6300.641420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.643087] Lustre: Skipped 1 previous similar message [ 6301.542635] Lustre: server umount lustre-MDT0000 complete [ 6305.115882] LustreError: 189699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041102 with bad export cookie 16460094223664160989 [ 6305.118887] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6305.142594] LustreError: 189699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6305.525664] Lustre: server umount lustre-MDT0001 complete [ 6313.701865] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:31:49 (1786041109) [ 6320.162478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.516441] LustreError: 193635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.574834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.542622] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6325.731933] LustreError: 193636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.520043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.954827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.541490] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6347.216804] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.563862] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6360.024795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.289611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.425947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6366.843977] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6375.332812] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6383.638792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.055328] Lustre: lustre-OST0001: new disk, initializing [ 6384.064827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6384.072312] Lustre: 196478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6384.208551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6391.923222] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6393.111564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6393.152114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6393.382786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6405.086166] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6408.676980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.682197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.704152] Lustre: Skipped 1 previous similar message [ 6408.718873] Lustre: Skipped 1 previous similar message [ 6414.795327] Lustre: server umount lustre-OST0000 complete [ 6416.095300] LustreError: 195457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.113250] LustreError: 195457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6420.862334] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6421.117565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6425.784905] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6425.993379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6434.275026] LustreError: 195455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.302887] LustreError: 195455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6444.517079] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.532977] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6444.538494] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6444.541059] Lustre: Skipped 3 previous similar messages [ 6444.556289] LustreError: Skipped 1 previous similar message [ 6446.061371] Lustre: server umount lustre-OST0001 complete [ 6451.681243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6451.689445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.702954] Lustre: Skipped 1 previous similar message [ 6456.227639] Lustre: server umount lustre-MDT0000 complete [ 6459.354316] LustreError: 196476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041256 with bad export cookie 16460094223664162060 [ 6459.366941] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.671226] Lustre: server umount lustre-MDT0001 complete [ 6486.669348] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:34:42 (1786041282) [ 6493.783185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6494.003682] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6494.040122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6494.217457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.183819] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6508.308657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.364479] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6508.392805] Lustre: 199917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6513.100710] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6520.712550] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6526.622886] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6533.350306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.653064] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6540.234493] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6542.993401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6547.575669] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.164246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6554.218935] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6562.178179] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6562.540162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6572.840817] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6578.159434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.189262] Lustre: Skipped 1 previous similar message [ 6578.208278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.226032] Lustre: Skipped 5 previous similar messages [ 6582.166352] Lustre: server umount lustre-OST0000 complete [ 6583.279525] LustreError: 201768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.299076] LustreError: 201768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6598.625197] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6598.635765] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.654805] Lustre: Skipped 2 previous similar messages [ 6604.545725] Lustre: server umount lustre-OST0001 complete [ 6613.986311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6613.991290] Lustre: Skipped 8 previous similar messages [ 6614.782738] Lustre: server umount lustre-MDT0000 complete [ 6618.311546] LustreError: 199909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041415 with bad export cookie 16460094223664163460 [ 6618.319235] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.325974] LustreError: 199909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6618.720302] Lustre: server umount lustre-MDT0001 complete [ 6641.389864] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:37:16 (1786041436) [ 6656.536923] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 6665.354711] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6710.096160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.324366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6710.352723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6710.424070] Lustre: lustre-MDT0000: new disk, initializing [ 6710.477059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.483610] Lustre: Skipped 3 previous similar messages [ 6710.499492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6715.207402] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6726.180037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.340444] Lustre: 208016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6726.395438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6726.400853] Lustre: Skipped 1 previous similar message [ 6726.464931] Lustre: lustre-MDT0001: new disk, initializing [ 6726.592930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6726.610524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.960462] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6741.513955] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.879982] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6756.514319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.956794] Lustre: lustre-OST0000: new disk, initializing [ 6756.964856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6756.984648] Lustre: 209981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6764.288423] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6767.126104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6767.144571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6767.213418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6774.386386] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.450070] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6780.748279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6785.757928] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6786.068642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6792.677488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.678773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.691332] LustreError: Skipped 1 previous similar message [ 6792.692666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6792.712380] Lustre: Skipped 3 previous similar messages [ 6795.131380] Lustre: server umount lustre-OST0000 complete [ 6806.466669] Lustre: server umount lustre-MDT0000 complete [ 6808.046534] LustreError: 209168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.072724] LustreError: 209168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6811.481259] LustreError: 208006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041608 with bad export cookie 16460094223664165245 [ 6811.490778] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.499641] LustreError: 208006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6812.581273] Lustre: server umount lustre-MDT0001 complete [ 6823.044621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.868439] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6839.081343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.425236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6839.442364] Lustre: Skipped 3 previous similar messages [ 6844.646210] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6853.347745] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.416551] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.638725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6875.029639] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6884.040824] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6889.656455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6889.893384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6894.811146] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6895.061933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6899.682248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6899.686468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.710554] LustreError: Skipped 1 previous similar message [ 6899.743430] Lustre: Skipped 3 previous similar messages [ 6903.396858] Lustre: server umount lustre-OST0000 complete [ 6908.981912] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6913.280937] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6942.496238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.627868] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 6956.765217] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.344613] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6964.215310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6964.223739] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6964.224809] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6964.250407] Lustre: Skipped 1 previous similar message [ 6969.322963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6969.325199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.330327] LustreError: Skipped 1 previous similar message [ 6969.336084] Lustre: Skipped 6 previous similar messages [ 6970.651648] Lustre: server umount lustre-OST0000 complete [ 6988.257504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.719592] Lustre: server umount lustre-MDT0000 complete [ 6989.797257] LustreError: 212199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.812754] LustreError: 212199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6992.181560] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041789 with bad export cookie 16460094223664166134 [ 6992.182370] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.192547] LustreError: 212179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6992.669556] Lustre: server umount lustre-MDT0001 complete [ 7001.425709] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:43:17 (1786041797) [ 7007.416170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.990662] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7020.618977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.681360] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7034.567808] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7042.194381] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7049.527208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.712493] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7064.688430] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.720377] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7068.845975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7073.530768] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7073.780339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7079.943333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.077568] Lustre: lustre-OST0001: new disk, initializing [ 7080.080885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7080.085931] Lustre: 221536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7085.813949] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7089.261157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7089.265063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7089.341796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7094.247569] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.918265] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7099.124760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7104.651650] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7104.943400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7110.121405] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.122357] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7110.131524] Lustre: Skipped 5 previous similar messages [ 7121.376088] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.481190] Lustre: server umount lustre-OST0001 complete [ 7127.085539] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7127.230911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7131.331239] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7131.505884] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7161.058491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7161.311973] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.318512] Lustre: Skipped 6 previous similar messages [ 7166.471279] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7166.507879] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7166.516882] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7166.542831] Lustre: Skipped 1 previous similar message [ 7168.180800] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7175.951110] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.513726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7181.806408] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7181.818835] LustreError: Skipped 1 previous similar message [ 7194.081337] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7194.182720] Lustre: server umount lustre-OST0001 complete [ 7217.881752] Lustre: server umount lustre-OST0000 complete [ 7227.878733] Lustre: server umount lustre-MDT0000 complete [ 7231.401279] LustreError: 218149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042028 with bad export cookie 16460094223664167100 [ 7231.413832] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.416108] LustreError: 218149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7237.819349] Lustre: server umount lustre-MDT0001 complete [ 7260.668719] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:47:36 (1786042056) [ 7269.504139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.745294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7269.776826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7275.219654] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7284.213890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.272837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7284.298321] Lustre: 226793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7289.894306] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7298.382259] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7304.620831] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7311.364794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7311.531349] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7317.827575] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7326.650604] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7335.808819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7335.845832] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7342.575863] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7350.632558] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7352.114371] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7356.744819] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7356.921494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7361.073531] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7361.311594] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7367.092935] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc9606ef800.ost_server_uuid 50 [ 7368.820352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc9606ef800.ost_server_uuid in IDLE state after 0 sec [ 7369.926269] Lustre: Permanently deactivating lustre-OST0001 [ 7369.961054] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7372.614666] Lustre: Permanently reactivating lustre-OST0001 [ 7378.409991] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.439569] Lustre: Skipped 6 previous similar messages [ 7378.449092] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7378.451874] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7378.461628] Lustre: Skipped 25 previous similar messages [ 7383.186899] Lustre: server umount lustre-OST0001 complete [ 7383.527682] LustreError: 229125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.538568] LustreError: 229125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 7393.671622] Lustre: server umount lustre-OST0000 complete [ 7404.036234] Lustre: server umount lustre-MDT0000 complete [ 7407.630319] LustreError: 226784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042204 with bad export cookie 16460094223664168381 [ 7407.636037] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7407.637101] LustreError: 226784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7408.020974] Lustre: server umount lustre-MDT0001 complete [ 7438.574642] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:50:33 (1786042233) [ 7447.834483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.120818] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7448.155734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7448.164524] Lustre: Skipped 5 previous similar messages [ 7452.934147] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7461.252726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.329735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7461.354695] Lustre: 233487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7465.716703] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7473.650813] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7479.328430] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7485.705981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.885691] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7485.906106] Lustre: Permanently deactivating lustre-OST0000 [ 7485.918855] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7492.366329] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7492.661413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7500.099991] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7507.253114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7507.307771] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7513.783627] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7514.844287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7521.628084] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.790951] Lustre: Permanently reactivating lustre-OST0000 [ 7523.817421] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7523.823295] Lustre: Skipped 2 previous similar messages [ 7533.650712] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7533.654081] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 000000009e8920a1, cur 1786042330 deadline 1786042319 last 1786042289 [ 7533.659685] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7533.676081] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7533.748287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7545.077605] Lustre: server umount lustre-OST0001 complete [ 7554.685391] Lustre: server umount lustre-OST0000 complete [ 7564.817539] Lustre: server umount lustre-MDT0000 complete [ 7568.097788] LustreError: 233480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042365 with bad export cookie 16460094223664170040 [ 7568.098696] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7568.106311] LustreError: 233480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7568.389681] Lustre: server umount lustre-MDT0001 complete [ 7580.757632] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 7583.297794] Key type lgssc unregistered [ 7583.643332] LNet: 238369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7583.663071] LNetError: 238369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7584.681622] LNet: Removed LNI 192.168.201.130@tcp [ 7585.391905] Key type .llcrypt unregistered [ 7585.397382] Key type ._llcrypt unregistered [ 7614.512953] Key type ._llcrypt registered [ 7614.515781] Key type .llcrypt registered [ 7614.613674] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7615.562195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7615.600799] alg: No test for adler32 (adler32-zlib) [ 7616.627563] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7616.793508] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 7618.472392] Key type lgssc registered [ 7620.003472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7629.698297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7629.722548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.192829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7636.947859] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7641.590812] LustreError: 239839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7641.628535] LustreError: 239839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7646.583489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.689979] LustreError: 239840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.765849] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7646.802781] Lustre: 239828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7646.815790] Lustre: Permanently deactivating lustre-MDT0001 [ 7646.835789] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7647.283580] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7647.294254] LustreError: 239870:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7647.317262] LustreError: 239870:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7647.352665] LustreError: 239870:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7647.458700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.587391] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7664.086225] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7671.508330] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7681.489074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.957937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7690.844472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7692.205576] LustreError: 241681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.381960] LustreError: 241683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.084200] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7701.484488] LustreError: 242404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.501249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7701.508443] LustreError: 242404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7706.413187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.620900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7711.740848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7711.749521] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7712.257804] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7718.917651] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7723.017720] Lustre: Permanently reactivating lustre-MDT0001 [ 7723.038791] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7723.043076] Lustre: Skipped 2 previous similar messages [ 7729.405543] Lustre: lustre-MDT0001: Client 9369bbe8-6a3f-4661-8379-2957edb1ef6c (at 192.168.201.30@tcp) reconnecting [ 7735.657491] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7737.725695] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7743.676955] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7743.898963] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7748.936723] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7749.179336] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7751.911565] Lustre: Permanently deactivating lustre-MDT0001 [ 7751.926308] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7751.929460] Lustre: Skipped 2 previous similar messages [ 7758.028090] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7764.964778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7764.968295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.977119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.678861] Lustre: server umount lustre-MDT0000 complete [ 7768.039989] LustreError: 239839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.051259] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7768.109724] Lustre: Skipped 1 previous similar message [ 7770.128879] LustreError: 243388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042567 with bad export cookie 10508724974139698355 [ 7770.139991] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.144111] LustreError: 243388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7770.443043] Lustre: server umount lustre-MDT0001 complete [ 7777.514351] Lustre: server umount lustre-OST0000 complete [ 7791.374350] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:56:27 (1786042587) [ 7803.290398] Lustre: server umount lustre-OST0001 complete [ 7809.884228] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 7820.697416] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 7856.927738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.436415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7857.453242] Lustre: Skipped 2 previous similar messages [ 7857.504610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7857.624949] Lustre: lustre-MDT0000: new disk, initializing [ 7857.729927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7857.758719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7862.790155] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7873.947376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.040750] Lustre: 248099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7874.085773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7874.092236] Lustre: Skipped 1 previous similar message [ 7874.222349] Lustre: lustre-MDT0001: new disk, initializing [ 7874.346608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7874.443535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7874.449416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7878.681221] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7888.006884] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7894.412122] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7901.745924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.047858] Lustre: lustre-OST0000: new disk, initializing [ 7902.052022] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7902.058773] Lustre: 250068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7902.140251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.285369] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7908.950755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7908.964407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7909.099172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7918.256354] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7920.506089] LustreError: 248106:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7929.382498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7929.580692] Lustre: lustre-OST0001: new disk, initializing [ 7929.583942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7929.594120] Lustre: 251184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7930.545280] LustreError: 248106:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7930.577284] LustreError: 248107:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7936.433658] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 7939.693407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7939.707817] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7939.854495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7940.608154] LustreError: 248107:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7947.062241] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7949.319248] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7955.433044] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7955.436996] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.449090] LustreError: Skipped 1 previous similar message [ 7955.457892] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7955.468661] Lustre: Skipped 2 previous similar messages [ 7955.492646] Lustre: Skipped 1 previous similar message [ 7960.563791] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7960.578737] Lustre: Skipped 1 previous similar message [ 7965.669889] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7965.675530] Lustre: Skipped 1 previous similar message [ 7966.176214] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7966.309594] Lustre: server umount lustre-OST0001 complete [ 7970.791428] LustreError: 250050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.828897] LustreError: 250050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7991.267370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.270710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7991.306721] Lustre: Skipped 1 previous similar message [ 7991.323413] Lustre: Skipped 1 previous similar message [ 7992.817262] Lustre: server umount lustre-OST0000 complete [ 7997.409870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7997.416568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.506251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8001.513079] Lustre: Skipped 2 previous similar messages [ 8002.909408] Lustre: server umount lustre-MDT0000 complete [ 8006.249322] LustreError: 248092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042803 with bad export cookie 10508724974139700952 [ 8006.261751] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8012.683336] Lustre: server umount lustre-MDT0001 complete [ 8024.933836] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8028.527718] Key type lgssc unregistered [ 8028.832494] LNet: 253220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8028.840821] LNetError: 253220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8028.852991] LNet: Removed LNI 192.168.201.130@tcp [ 8029.579268] Key type .llcrypt unregistered [ 8029.584354] Key type ._llcrypt unregistered [ 8070.729979] Key type ._llcrypt registered [ 8070.732989] Key type .llcrypt registered [ 8070.792553] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8072.435235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8072.471035] alg: No test for adler32 (adler32-zlib) [ 8073.573899] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8073.861627] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8075.490879] Key type lgssc registered [ 8077.283662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8086.361707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8086.382573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.810624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8087.865395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8088.163584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.233944] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8104.931356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.092933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8105.139175] Lustre: 255442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8105.469700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8111.015919] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8118.892717] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8126.963591] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8134.200786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.491800] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8134.637870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8139.902661] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8141.589573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8148.707761] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8203.244679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8203.251252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.252171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8203.267112] LustreError: Skipped 1 previous similar message [ 8203.336990] Lustre: Skipped 1 previous similar message [ 8207.640825] Lustre: server umount lustre-OST0000 complete [ 8212.159096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8227.446276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.833148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.614790] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8242.406421] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8243.196017] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8243.221429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8243.222664] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8284.136531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.152602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.156046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.165258] Lustre: Skipped 2 previous similar messages [ 8289.251311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8289.260222] Lustre: Skipped 1 previous similar message [ 8289.426160] Lustre: server umount lustre-OST0000 complete [ 8294.372090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.387378] Lustre: Skipped 1 previous similar message [ 8294.392311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8294.400310] Lustre: Skipped 1 previous similar message [ 8299.495811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.512339] Lustre: Skipped 1 previous similar message [ 8301.755402] Lustre: server umount lustre-MDT0000 complete [ 8305.678137] LustreError: 255435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043102 with bad export cookie 1169996514753100532 [ 8305.680964] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8305.690657] LustreError: 255435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8305.982258] Lustre: server umount lustre-MDT0001 complete [ 8317.950326] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8321.146949] Key type lgssc unregistered [ 8321.452655] LNet: 260873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8321.463690] LNetError: 260873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8321.481305] LNet: Removed LNI 192.168.201.130@tcp [ 8322.441092] Key type .llcrypt unregistered [ 8322.445909] Key type ._llcrypt unregistered [ 8348.324730] Key type ._llcrypt registered [ 8348.328456] Key type .llcrypt registered [ 8348.413765] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8349.480143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8349.651068] alg: No test for adler32 (adler32-zlib) [ 8350.652760] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8350.861260] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8352.512232] Key type lgssc registered [ 8353.598318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8360.811783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8360.822975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.378607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.035720] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8372.714451] LustreError: 262293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.733051] LustreError: 262293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8376.172986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.585603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.692490] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8391.755258] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8397.927785] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8407.290286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8407.530701] LustreError: 264112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.882194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.169337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8414.074892] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8421.988331] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.951678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.954958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8446.964805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8446.975962] Lustre: Skipped 1 previous similar message [ 8450.963251] Lustre: server umount lustre-OST0000 complete [ 8457.185583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.206279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.216549] Lustre: Skipped 2 previous similar messages [ 8460.856585] Lustre: server umount lustre-MDT0000 complete [ 8462.305782] LustreError: 262293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.321526] LustreError: 262293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8464.110360] LustreError: 264136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043261 with bad export cookie 9624279083343871529 [ 8464.118515] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.120327] LustreError: 264136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8464.693498] Lustre: server umount lustre-MDT0001 complete [ 8477.182477] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8479.769698] Key type lgssc unregistered [ 8480.044432] LNet: 266602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.052407] LNetError: 266602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8481.126879] LNet: Removed LNI 192.168.201.130@tcp [ 8481.832699] Key type .llcrypt unregistered [ 8481.835571] Key type ._llcrypt unregistered [ 8503.047799] Key type ._llcrypt registered [ 8503.050378] Key type .llcrypt registered [ 8503.168448] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8503.861771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8504.068242] alg: No test for adler32 (adler32-zlib) [ 8505.140628] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8505.437953] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8507.128161] Key type lgssc registered [ 8508.792702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8517.634428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8517.666269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.379794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.447491] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8529.891956] LustreError: 267827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.935362] LustreError: 267827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8535.026030] LustreError: 267828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.132528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.593902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8539.918253] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8547.960977] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8554.313924] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8561.793840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.541040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8564.604680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8570.943663] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8580.094396] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8600.545288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8600.561382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.563074] LustreError: Skipped 1 previous similar message [ 8600.566286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.607387] Lustre: Skipped 1 previous similar message [ 8605.665867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8605.668212] Lustre: Skipped 1 previous similar message [ 8610.272196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8610.493525] Lustre: server umount lustre-OST0000 complete [ 8615.905214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.906941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.927227] Lustre: Skipped 1 previous similar message [ 8615.944415] Lustre: Skipped 2 previous similar messages [ 8621.026504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8621.040246] Lustre: Skipped 1 previous similar message [ 8621.943572] Lustre: server umount lustre-MDT0000 complete [ 8626.146638] LustreError: 267823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.173800] LustreError: 267823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8626.431575] LustreError: 270551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043423 with bad export cookie 4894762861345332748 [ 8626.438824] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8626.440236] LustreError: 270551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8626.909972] Lustre: server umount lustre-MDT0001 complete [ 8639.478063] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8642.619817] Key type lgssc unregistered [ 8642.970663] LNet: 271510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8642.981521] LNetError: 271510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8643.018610] LNet: Removed LNI 192.168.201.130@tcp [ 8643.956439] Key type .llcrypt unregistered [ 8643.958596] Key type ._llcrypt unregistered [ 8673.664079] Key type ._llcrypt registered [ 8673.667524] Key type .llcrypt registered [ 8673.844437] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8675.428975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8675.481030] alg: No test for adler32 (adler32-zlib) [ 8676.601806] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8676.845243] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8678.632261] Key type lgssc registered [ 8679.890180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8688.263927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8688.294733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.829057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8694.216438] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8699.371128] LustreError: 272930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.385318] LustreError: 272930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8703.757597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.207512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.285598] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8717.834118] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8724.103667] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8731.274413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.598788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.705977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8738.274876] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8747.244386] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8779.753219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.760855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8779.767630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8783.914713] Lustre: server umount lustre-OST0000 complete [ 8789.988083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.994896] Lustre: Skipped 1 previous similar message [ 8789.998161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8790.002549] Lustre: Skipped 1 previous similar message [ 8793.942287] Lustre: server umount lustre-MDT0000 complete [ 8795.109722] LustreError: 272931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.131690] LustreError: 272931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8798.897392] LustreError: 272911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043595 with bad export cookie 545739525592897606 [ 8798.911092] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8798.919494] LustreError: 272911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8799.260793] Lustre: server umount lustre-MDT0001 complete [ 8812.902563] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8816.224458] Key type lgssc unregistered [ 8816.552167] LNet: 277240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8816.555853] LNetError: 277240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8817.573601] LNet: Removed LNI 192.168.201.130@tcp [ 8818.458896] Key type .llcrypt unregistered [ 8818.461610] Key type ._llcrypt unregistered [ 8839.380587] Key type ._llcrypt registered [ 8839.383546] Key type .llcrypt registered [ 8839.529422] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 8840.881860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8840.893025] alg: No test for adler32 (adler32-zlib) [ 8842.101501] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8842.472536] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 8844.168765] Key type lgssc registered [ 8845.440974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8853.982800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8854.008134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.560064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.536484] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8865.765691] LustreError: 278477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.798025] LustreError: 278477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8869.018628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.425560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.657550] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8881.620781] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8888.098943] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8895.602178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8895.935719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.021947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8904.349703] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 8916.105635] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8934.883980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8934.899760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.905352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.932963] Lustre: Skipped 1 previous similar message [ 8938.592994] Lustre: server umount lustre-OST0000 complete [ 8945.131107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8945.137475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8945.153538] Lustre: Skipped 1 previous similar message [ 8945.173945] Lustre: Skipped 2 previous similar messages [ 8950.241856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8950.248471] Lustre: Skipped 1 previous similar message [ 8955.363958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.410319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8957.954210] Lustre: server umount lustre-MDT0000 complete [ 8960.481896] LustreError: 278480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.492787] LustreError: 278480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8962.317856] LustreError: 278443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043759 with bad export cookie 16229831392145254056 [ 8962.330725] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.664649] Lustre: server umount lustre-MDT0001 complete [ 8975.531904] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 8978.340570] Key type lgssc unregistered [ 8978.687708] LNet: 282183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8978.698771] LNetError: 282183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8978.731110] LNet: Removed LNI 192.168.201.130@tcp [ 8979.672822] Key type .llcrypt unregistered [ 8979.674806] Key type ._llcrypt unregistered [ 8993.179457] Key type ._llcrypt registered [ 8993.188539] Key type .llcrypt registered [ 8993.307248] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9011.974154] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9029.274868] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9030.265926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9030.319908] alg: No test for adler32 (adler32-zlib) [ 9031.475502] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9031.678942] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9033.353574] Key type lgssc registered [ 9034.430807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9073.809319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9073.846599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.250777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9075.287187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9075.413902] Lustre: lustre-MDT0000: new disk, initializing [ 9075.471170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.506843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9080.584302] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9090.936446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.080593] Lustre: 285740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9091.183083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9091.189343] Lustre: Skipped 1 previous similar message [ 9091.264584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9091.327962] Lustre: lustre-MDT0001: new disk, initializing [ 9091.444886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9091.505555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9091.516071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9096.879490] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9106.232571] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.411904] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.054559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9119.436471] Lustre: lustre-OST0000: new disk, initializing [ 9119.444634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9119.455386] Lustre: 287705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9119.609897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9126.476567] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9129.057665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9129.076035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9129.205911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9136.855564] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9141.532983] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9141.716646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9146.804181] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9146.961924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9149.924232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.930563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9149.973892] LustreError: Skipped 1 previous similar message [ 9149.981095] Lustre: Skipped 1 previous similar message [ 9149.994364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9150.012056] Lustre: Skipped 1 previous similar message [ 9155.050277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9155.790324] Lustre: server umount lustre-OST0000 complete [ 9160.165531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.168185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.186099] Lustre: Skipped 1 previous similar message [ 9160.202280] Lustre: Skipped 2 previous similar messages [ 9165.282062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9165.292625] Lustre: Skipped 1 previous similar message [ 9166.539098] Lustre: server umount lustre-MDT0000 complete [ 9171.302424] LustreError: 285730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043968 with bad export cookie 10981917854554890493 [ 9171.320257] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9171.684314] Lustre: server umount lustre-MDT0001 complete [ 9181.175242] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:19:36 (1786043976) [ 9188.864618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.420730] LustreError: 290118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.443187] LustreError: 290118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9189.565850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.983256] LustreError: 290119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9195.175305] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9199.076183] LustreError: 290118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.204685] LustreError: 290119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.490708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9205.908991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9211.870498] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9219.559775] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9227.483141] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9236.714779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9237.191721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.642391] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9252.058238] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9275.366449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9275.368361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9275.375276] Lustre: Skipped 1 previous similar message [ 9275.376405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9275.422306] Lustre: Skipped 1 previous similar message [ 9279.565771] Lustre: server umount lustre-OST0000 complete [ 9285.561166] Lustre: server umount lustre-MDT0000 complete [ 9285.602428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.619079] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.630515] LustreError: 290113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9289.216463] LustreError: 290100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044086 with bad export cookie 10981917854554891375 [ 9289.222507] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9289.235291] LustreError: 290100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9289.704479] Lustre: server umount lustre-MDT0001 complete [ 9302.115808] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 9304.708338] Key type lgssc unregistered [ 9305.025663] LNet: 293679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9305.037744] LNetError: 293679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9305.062153] LNet: Removed LNI 192.168.201.130@tcp [ 9305.826174] Key type .llcrypt unregistered [ 9305.828864] Key type ._llcrypt unregistered [ 9335.125321] Key type ._llcrypt registered [ 9335.134560] Key type .llcrypt registered [ 9335.285832] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9354.731302] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9355.842323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9356.006493] alg: No test for adler32 (adler32-zlib) [ 9357.142822] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9357.414249] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9359.072176] Key type lgssc registered [ 9360.495909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9414.365627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9414.384292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9415.780645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9415.840303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9416.073217] Lustre: lustre-MDT0000: new disk, initializing [ 9416.206361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9416.230809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9420.993755] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9431.815503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.957857] Lustre: 297508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9432.048680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9432.053564] Lustre: Skipped 1 previous similar message [ 9432.218431] Lustre: lustre-MDT0001: new disk, initializing [ 9432.359871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9432.423182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9432.442699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9437.615485] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9448.697611] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9456.992640] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9467.013491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.801763] Lustre: lustre-OST2710: new disk, initializing [ 9467.812996] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9467.842781] Lustre: 299477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9468.063937] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9475.114568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9475.122708] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9475.238244] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9477.622317] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9489.903505] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9498.686275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.879794] Lustre: lustre-OST03e8: new disk, initializing [ 9498.888037] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9498.896893] Lustre: 300542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9498.991095] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9505.876703] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9508.435273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9508.443937] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9508.536641] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9515.539584] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9526.169881] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9526.525380] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9532.950366] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9533.288676] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9539.282617] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9539.668625] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9544.643177] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9544.776878] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9550.979039] Lustre: server umount lustre-MDT0000 complete [ 9554.917361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.944337] Lustre: Skipped 3 previous similar messages [ 9555.271487] LustreError: 299475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044352 with bad export cookie 8111378597929748071 [ 9555.289720] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.714193] Lustre: server umount lustre-MDT0001 complete [ 9569.348062] Lustre: server umount lustre-OST2710 complete [ 9584.024172] Lustre: server umount lustre-OST03e8 complete [ 9602.274472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 9614.173805] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9648.885271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.083680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9649.109784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9649.174188] Lustre: lustre-MDT0000: new disk, initializing [ 9649.253989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.269869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9652.988536] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9664.219604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.330629] Lustre: 305791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9664.383304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9664.386660] Lustre: Skipped 1 previous similar message [ 9664.450259] Lustre: lustre-MDT0001: new disk, initializing [ 9664.532342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.574936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9664.598889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9669.988886] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9681.756684] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9688.201683] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9694.876076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9695.234335] Lustre: lustre-OST0000: new disk, initializing [ 9695.236610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9695.242555] Lustre: 307762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9695.402600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9701.621995] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9701.957071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9701.970421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9702.101829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9710.609532] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9715.303089] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9715.511750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9719.615408] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9719.808044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9722.857587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.863562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.863724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.876222] Lustre: Skipped 1 previous similar message [ 9722.896967] LustreError: Skipped 1 previous similar message [ 9728.036550] Lustre: server umount lustre-OST0000 complete [ 9733.092073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.099798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9733.105320] Lustre: Skipped 1 previous similar message [ 9733.121989] Lustre: Skipped 2 previous similar messages [ 9737.868422] Lustre: server umount lustre-MDT0000 complete [ 9738.210192] LustreError: 306636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.222517] LustreError: 306636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9741.288137] LustreError: 305785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044538 with bad export cookie 8111378597929749751 [ 9741.293870] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9741.302861] LustreError: 305785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9741.623772] Lustre: server umount lustre-MDT0001 complete [ 9749.765865] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:29:05 (1786044545) [ 9751.199888] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9752.845901] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:29:08 (1786044548) [ 9756.964341] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9780.673268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9780.906798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9780.931513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9781.119393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.931857] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9794.190376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.370040] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9794.414718] Lustre: 311620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9799.421638] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9822.014962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.375667] Lustre: Denying initial registration attempt from nid 192.168.201.130@tcp, specified as failover [ 9822.381627] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9822.385931] LustreError: 313075:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9822.390550] LustreError: 313075:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9822.394354] LustreError: 313075:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9822.470727] Lustre: server umount lustre-OST0000 complete [ 9822.476703] LustreError: 313075:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9830.369301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.374542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.382940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9835.550739] Lustre: server umount lustre-MDT0000 complete [ 9837.038451] LustreError: 311633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.053807] LustreError: 311633:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9839.906551] LustreError: 311612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044636 with bad export cookie 8111378597929750633 [ 9839.913044] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.924613] LustreError: 311612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9840.187474] Lustre: server umount lustre-MDT0001 complete [ 9853.040826] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 9855.465895] Key type lgssc unregistered [ 9855.708751] LNet: 314199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9855.721319] LNetError: 314199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9855.736384] LNet: Removed LNI 192.168.201.130@tcp [ 9856.601179] Key type .llcrypt unregistered [ 9856.605537] Key type ._llcrypt unregistered [ 9870.615772] Key type ._llcrypt registered [ 9870.618944] Key type .llcrypt registered [ 9870.724207] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9872.013117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9872.037966] alg: No test for adler32 (adler32-zlib) [ 9873.052595] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9873.342212] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 9875.056227] Key type lgssc registered [ 9876.044593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9924.923254] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 9934.504750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9934.522539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9935.975051] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9936.015583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9936.192352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9940.869572] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9950.985736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9951.188235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9951.289752] Lustre: 316965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9951.659288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9956.570440] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 9964.821001] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9972.365321] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9992.824996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.182389] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9993.424651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10000.061356] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10011.745372] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10017.772895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10017.774415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.789785] LustreError: Skipped 1 previous similar message [10017.801358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10017.821784] Lustre: Skipped 1 previous similar message [10021.711979] Lustre: server umount lustre-OST0000 complete [10028.006395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.018945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.024558] Lustre: Skipped 1 previous similar message [10033.121299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.285101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.292026] Lustre: Skipped 2 previous similar messages [10040.289762] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10040.436551] Lustre: server umount lustre-MDT0000 complete [10046.039546] LustreError: 318837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044843 with bad export cookie 39567474004870944 [10046.065265] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10046.091761] LustreError: 318837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10046.639336] Lustre: server umount lustre-MDT0001 complete [10060.456303] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [10062.908646] Key type lgssc unregistered [10063.158820] LNet: 320549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10063.162996] LNetError: 320549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10063.172610] LNet: Removed LNI 192.168.201.130@tcp [10063.847596] Key type .llcrypt unregistered [10063.849569] Key type ._llcrypt unregistered [10091.288518] Key type ._llcrypt registered [10091.293381] Key type .llcrypt registered [10091.426232] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10092.527478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10092.642255] alg: No test for adler32 (adler32-zlib) [10093.944801] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [10094.261109] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [10095.920278] Key type lgssc registered [10097.430927] Lustre: Echo OBD driver; http://www.lustre.org/ [10105.883020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10105.923983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.471304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.207548] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10117.606159] LustreError: 321964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.615077] LustreError: 321964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10122.723650] LustreError: 321965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.855544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.262961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.441922] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10139.142439] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10146.703322] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10157.106179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10157.533860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.945520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10174.048372] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10190.569709] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.30@tcp (stopping) [10190.818280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10190.825562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.913712] Lustre: server umount lustre-MDT0000 complete [10192.358718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.364982] LustreError: 321964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.378636] Lustre: Skipped 1 previous similar message [10192.428664] LustreError: 321964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10195.653782] LustreError: 321961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.661764] LustreError: 321947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044993 with bad export cookie 890163711789060645 [10196.682385] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.482742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.506287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10197.515330] Lustre: Skipped 1 previous similar message [10200.791143] LustreError: 324459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.827439] LustreError: 324459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10200.847843] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [10203.508901] Lustre: server umount lustre-MDT0001 complete [10211.601107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10222.971542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.507436] LustreError: 321274:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007160a55e x1872803807666560/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10241.738214] LustreError: 325585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.783150] LustreError: 325585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10242.103258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10242.194888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10247.923445] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10256.378905] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10257.933390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.383697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10263.544862] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10264.007889] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10273.564388] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10275.630923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10282.119228] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10284.304645] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10293.220339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10293.234308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.253700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10293.263996] Lustre: Skipped 1 previous similar message [10294.242867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10296.024562] Lustre: server umount lustre-OST0000 complete [10304.481612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10304.486121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.502692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10304.508504] Lustre: Skipped 2 previous similar messages [10304.520197] Lustre: Skipped 3 previous similar messages [10306.948493] Lustre: server umount lustre-MDT0000 complete [10309.602803] LustreError: 325591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.621707] LustreError: 325591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10311.578031] LustreError: 321947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045108 with bad export cookie 890163711789068086 [10311.590689] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10311.597084] LustreError: 321947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10311.968043] Lustre: server umount lustre-MDT0001 complete [10325.803313] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [10329.750633] Key type lgssc unregistered [10330.127327] LNet: 328263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10330.134637] LNetError: 328263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10330.165738] LNet: Removed LNI 192.168.201.130@tcp [10331.607227] Key type .llcrypt unregistered [10331.618674] Key type ._llcrypt unregistered [10363.771664] Key type ._llcrypt registered [10363.775576] Key type .llcrypt registered [10363.927298] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10365.636404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10365.749791] alg: No test for adler32 (adler32-zlib) [10366.884156] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [10367.213737] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [10368.992349] Key type lgssc registered [10370.237679] Lustre: Echo OBD driver; http://www.lustre.org/ [10378.040218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10378.062907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.763636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.661421] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10388.972081] LustreError: 329682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.999949] LustreError: 329682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10394.088561] LustreError: 329683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.511851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.130447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10402.085819] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10410.346575] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10417.802978] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10421.735690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10421.739388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10421.784055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10422.820814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10422.838077] Lustre: Skipped 2 previous similar messages [10427.670046] Lustre: server umount lustre-MDT0000 complete [10427.890550] LustreError: 329682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10427.928717] LustreError: 329682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10432.379571] LustreError: 329664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045229 with bad export cookie 6667430802028747425 [10432.386801] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10432.406146] LustreError: 329664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10433.042610] Lustre: server umount lustre-MDT0001 complete [10449.044387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.426579] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10449.481458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10449.685324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.751218] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10465.015186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.116369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10465.154688] Lustre: 332269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10465.480330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10470.099423] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10478.396191] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10484.531461] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10494.902339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10495.210425] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10495.232879] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10495.260347] LustreError: 334155:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10495.271629] LustreError: 334155:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10495.278212] LustreError: 334155:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10495.370088] Lustre: server umount lustre-OST0000 complete [10495.372778] LustreError: 334155:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10507.317082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10507.626300] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10507.909880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10514.760840] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10517.232073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10523.956861] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10533.252773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10533.334848] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10533.430715] Lustre: lustre-OST0001: new disk, initializing [10533.444393] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10533.456713] Lustre: 335697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10533.590928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10540.181275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10540.201563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10540.283196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10540.633580] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10549.541534] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10555.880458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10555.883401] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.886679] LustreError: Skipped 1 previous similar message [10555.888755] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10555.897713] Lustre: Skipped 2 previous similar messages [10555.910147] Lustre: Skipped 1 previous similar message [10558.808620] Lustre: server umount lustre-OST0001 complete [10561.016589] LustreError: 334621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.046201] LustreError: 334621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10563.554305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10563.571604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.601061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10569.193762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10569.198208] LustreError: 334621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.209231] Lustre: Skipped 3 previous similar messages [10569.273577] LustreError: 334621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10569.791332] Lustre: server umount lustre-OST0000 complete [10578.409813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10578.416261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.427222] Lustre: Skipped 1 previous similar message [10578.442710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.673491] Lustre: server umount lustre-MDT0000 complete [10584.570587] LustreError: 332283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10584.592645] LustreError: 332283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10588.762376] LustreError: 332264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045385 with bad export cookie 6667430802028747978 [10588.765212] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.774220] LustreError: 332264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10589.336070] Lustre: server umount lustre-MDT0001 complete [10602.268128] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [10606.243217] Key type lgssc unregistered [10606.844890] LNet: 337660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10606.854909] LNetError: 337660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10606.881709] LNet: Removed LNI 192.168.201.130@tcp [10608.016741] Key type .llcrypt unregistered [10608.019279] Key type ._llcrypt unregistered [10694.556861] Key type ._llcrypt registered [10694.559508] Key type .llcrypt registered [10694.692386] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [10712.014627] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [10713.077818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10713.125241] alg: No test for adler32 (adler32-zlib) [10714.289197] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [10714.546772] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [10716.232980] Key type lgssc registered [10717.530751] Lustre: Echo OBD driver; http://www.lustre.org/ [10756.788117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.823692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.181523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10758.210239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10758.267454] Lustre: lustre-MDT0000: new disk, initializing [10758.353072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10758.390082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10762.955560] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10773.995596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.137938] Lustre: 342601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10774.194755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10774.198756] Lustre: Skipped 1 previous similar message [10774.288233] Lustre: lustre-MDT0001: new disk, initializing [10774.461385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10774.496813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10774.519052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10779.799717] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10789.542604] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10795.905838] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10803.431426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10803.709802] Lustre: lustre-OST0000: new disk, initializing [10803.713359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10803.723598] Lustre: 344569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10803.833634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10811.119104] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10813.550957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10813.560208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10813.658158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10821.807716] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10827.388494] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10827.671919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10834.090983] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10834.466469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10839.522658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.530921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10839.550740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10839.574926] Lustre: Skipped 1 previous similar message [10842.719785] Lustre: server umount lustre-OST0000 complete [10849.762927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.781232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.794099] Lustre: Skipped 2 previous similar messages [10853.438076] Lustre: server umount lustre-MDT0000 complete [10857.005079] LustreError: 342592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045654 with bad export cookie 5880595566649553914 [10857.017142] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10857.024574] LustreError: 342592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10857.358676] Lustre: server umount lustre-MDT0001 complete [10867.361633] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:47:42 (1786045662) [10876.058876] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:47:51 (1786045671) [10885.036525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.479380] LustreError: 347326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.499583] LustreError: 347326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10885.573934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10889.905221] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10890.721771] LustreError: 347327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.843057] LustreError: 347326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10898.156274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.485890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10903.341534] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10911.867078] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10918.480980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10927.385478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.628902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10934.830736] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [10943.606648] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10951.568441] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10953.598570] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10955.435415] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10957.879885] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10960.201022] Lustre: Failing over lustre-MDT0000 [10960.564654] Lustre: server umount lustre-MDT0000 complete [10960.631513] LustreError: 347322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.429362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.449651] Lustre: Skipped 2 previous similar messages [10965.697593] LustreError: 347321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.716595] LustreError: 347321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10975.946817] LustreError: 349166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10975.968619] LustreError: 349166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10979.808487] Lustre: 340821:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786045760/real 1786045760] req@0000000027d16f8a x1872804457816448/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786045776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10979.847737] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10981.507798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.053069] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x519c1282d9e23b86 [10989.072335] Lustre: MGC192.168.201.130@tcp: Connection restored to 0@lo (at 0@lo) [10989.324651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.377750] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10991.315967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10994.671824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10994.697690] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10994.758920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10995.220924] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11006.328455] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11008.464504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11012.475855] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11014.631287] Lustre: Failing over lustre-MDT0000 [11014.919897] Lustre: server umount lustre-MDT0000 complete [11015.154583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.163865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11015.165424] LustreError: 348053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.165433] LustreError: 348053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [11015.181103] Lustre: Skipped 2 previous similar messages [11031.520103] Lustre: 340823:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786045812/real 1786045812] req@000000004ad7f4cb x1872804457837952/t0(0) o400->MGC192.168.201.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786045828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11031.531381] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.258997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.223567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.284708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11043.546333] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11047.435142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11047.446184] Lustre: Skipped 2 previous similar messages [11047.494489] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11047.573114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11048.795415] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11063.685391] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11066.684834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11078.131243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11078.133675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11078.167880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11078.174662] Lustre: Skipped 3 previous similar messages [11079.476056] Lustre: server umount lustre-MDT0000 complete [11083.235294] LustreError: 350921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11083.255681] LustreError: 350921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [11083.315803] LustreError: 347307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045880 with bad export cookie 5880595566649556630 [11083.317858] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.335546] LustreError: 347307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11083.985844] Lustre: server umount lustre-MDT0001 complete [11098.549595] Lustre: server umount lustre-OST0000 complete [11114.779936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.228740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11119.631441] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11128.784138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.271455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11140.709472] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11146.896120] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11153.915947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11159.923147] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11160.317110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11166.934789] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11175.904956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11175.907124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11175.927551] Lustre: Skipped 3 previous similar messages [11175.928421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11175.938421] Lustre: Skipped 1 previous similar message [11179.495926] Lustre: server umount lustre-OST0000 complete [11185.635117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11185.698931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11190.329477] Lustre: server umount lustre-MDT0000 complete [11191.265332] LustreError: 353676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.305282] LustreError: 353676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11193.998592] LustreError: 355518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786045991 with bad export cookie 5880595566649557379 [11194.000080] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11194.008891] LustreError: 355518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11194.304397] Lustre: server umount lustre-MDT0001 complete [11205.097661] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11207.475478] Key type lgssc unregistered [11207.699952] LNet: 357211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11207.708992] LNetError: 357211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11207.730252] LNet: Removed LNI 192.168.201.130@tcp [11208.745080] Key type .llcrypt unregistered [11208.747715] Key type ._llcrypt unregistered [11233.959465] Key type ._llcrypt registered [11233.961963] Key type .llcrypt registered [11234.055168] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11234.963091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11234.996410] alg: No test for adler32 (adler32-zlib) [11236.054356] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [11236.262074] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11237.936312] Key type lgssc registered [11238.899645] Lustre: Echo OBD driver; http://www.lustre.org/ [11245.487701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11245.517632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.837937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11251.331930] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11256.292721] LustreError: 358729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11256.310394] LustreError: 358729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11259.161526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.430858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11263.450592] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11272.358527] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11279.205420] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11286.685958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11287.064762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11291.188247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11293.528769] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11301.672991] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11320.806390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11320.807918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11320.814588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11320.833262] Lustre: Skipped 2 previous similar messages [11323.876812] Lustre: server umount lustre-MDT0000 complete [11325.924904] LustreError: 358728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.796812] LustreError: 360574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046124 with bad export cookie 1110775792032260443 [11327.800067] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.807219] LustreError: 360574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11328.261901] Lustre: server umount lustre-MDT0001 complete [11342.383162] Lustre: server umount lustre-OST0000 complete [11354.128059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11354.602700] LustreError: 362463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11354.630879] LustreError: 362463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11354.677870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.341916] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11359.718277] LustreError: 362464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11364.848205] LustreError: 362463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.626978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.972925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11372.192470] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11379.925790] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11385.922147] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11393.327164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.725051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11399.849796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11400.139707] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11407.658407] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11419.105617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11419.117886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.142057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11419.146886] Lustre: Skipped 2 previous similar messages [11422.697218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.699581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11422.721879] Lustre: Skipped 2 previous similar messages [11425.498603] Lustre: server umount lustre-MDT0000 complete [11427.811469] LustreError: 362458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.852636] LustreError: 362458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11429.526314] LustreError: 363198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046226 with bad export cookie 1110775792032261556 [11429.548428] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11430.066120] Lustre: server umount lustre-MDT0001 complete [11445.099237] Lustre: server umount lustre-OST0000 complete [11459.586661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.979864] LustreError: 366373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.083783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11464.782699] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11473.318388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.111956] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11487.632430] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11494.283969] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11500.661523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.041384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11501.046661] Lustre: Skipped 1 previous similar message [11505.136375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11507.571799] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11514.522305] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11520.484171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11520.489120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.496761] Lustre: Skipped 1 previous similar message [11520.500105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11523.555021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11525.791604] Lustre: server umount lustre-OST0000 complete [11530.209974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11530.226996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.237185] Lustre: Skipped 1 previous similar message [11530.244758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11530.249716] Lustre: Skipped 2 previous similar messages [11536.431238] Lustre: server umount lustre-MDT0000 complete [11538.917707] LustreError: 366373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11538.957512] LustreError: 366373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11542.040192] LustreError: 366354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046339 with bad export cookie 1110775792032262578 [11542.057487] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11542.088455] LustreError: 366354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11542.687953] Lustre: server umount lustre-MDT0001 complete [11557.678676] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11561.274351] Key type lgssc unregistered [11561.656661] LNet: 369907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11561.666990] LNetError: 369907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11562.734498] LNet: Removed LNI 192.168.201.130@tcp [11563.591170] Key type .llcrypt unregistered [11563.593977] Key type ._llcrypt unregistered [11589.755376] Key type ._llcrypt registered [11589.757966] Key type .llcrypt registered [11589.853170] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11590.749115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11590.784226] alg: No test for adler32 (adler32-zlib) [11591.964078] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [11592.271489] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11593.936308] Key type lgssc registered [11595.068146] Lustre: Echo OBD driver; http://www.lustre.org/ [11602.642901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11602.659059] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.664991] LustreError: 371353:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11602.678306] LustreError: 371353:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11602.688758] LustreError: 371353:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11602.694467] LustreError: 371353:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11602.699429] LustreError: 371353:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11609.686173] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11609.691184] LustreError: 371751:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11609.712308] LustreError: 371751:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11609.716745] LustreError: 371751:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11609.726703] LustreError: 371751:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11609.735631] LustreError: 371751:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11630.364217] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [11632.970816] Key type lgssc unregistered [11633.252704] LNet: 372757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11633.258460] LNetError: 372757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11633.284840] LNet: Removed LNI 192.168.201.130@tcp [11633.953944] Key type .llcrypt unregistered [11633.957166] Key type ._llcrypt unregistered [11653.789314] Key type ._llcrypt registered [11653.793861] Key type .llcrypt registered [11653.973904] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [11675.631095] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [11677.800678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11677.836301] alg: No test for adler32 (adler32-zlib) [11679.084748] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [11679.460916] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [11681.160312] Key type lgssc registered [11682.608036] Lustre: Echo OBD driver; http://www.lustre.org/ [11717.765821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11717.800821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.063890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11719.097259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11719.207942] Lustre: lustre-MDT0000: new disk, initializing [11719.300434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11719.320743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11723.605830] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11733.263712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11733.382426] Lustre: 375928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11733.425295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11733.428725] Lustre: Skipped 1 previous similar message [11733.485452] Lustre: lustre-MDT0001: new disk, initializing [11733.562125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11733.603810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11733.619755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11738.292705] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11748.175397] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11753.964897] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11760.611973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.934180] Lustre: lustre-OST0000: new disk, initializing [11760.937777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11760.950289] Lustre: 377896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11761.102582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11768.320898] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11770.469593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11770.480349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11770.601030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11778.369544] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11783.854120] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11784.055775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11788.552666] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11788.776577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11791.334748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11791.340249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11791.343525] LustreError: Skipped 1 previous similar message [11791.345234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11791.381099] Lustre: Skipped 1 previous similar message [11796.477310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11796.489705] Lustre: Skipped 2 previous similar messages [11797.757287] Lustre: server umount lustre-OST0000 complete [11801.570201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.590621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11801.600566] Lustre: Skipped 1 previous similar message [11806.694281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.698825] Lustre: Skipped 1 previous similar message [11807.992716] Lustre: server umount lustre-MDT0000 complete [11811.927254] LustreError: 377895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046608 with bad export cookie 12181113124001186968 [11811.930913] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11811.944278] LustreError: 377895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11812.399372] Lustre: server umount lustre-MDT0001 complete [11821.681632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11823.769761] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:03:39 (1786046619) [11830.570477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.160423] LustreError: 380355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.183610] LustreError: 380355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11831.294988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.400973] LustreError: 380356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.566838] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11841.519313] LustreError: 380355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.632446] LustreError: 380356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.818614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.255437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11854.121626] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11862.833095] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11869.097814] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11876.631380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11876.773477] LustreError: 382177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.052608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.360464] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11891.290878] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11899.608484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11899.766472] Lustre: lustre-OST0001: new disk, initializing [11899.769904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11899.776345] Lustre: 383195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11899.866712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11906.539855] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [11909.294711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11909.301243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11909.423153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11918.345696] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11924.961990] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11924.968989] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.971316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11924.983813] Lustre: Skipped 2 previous similar messages [11925.001345] Lustre: Skipped 1 previous similar message [11929.472493] Lustre: server umount lustre-OST0001 complete [11930.090369] LustreError: 382179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.111949] LustreError: 382179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11949.027128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11949.030173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.036886] LustreError: Skipped 2 previous similar messages [11949.047830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.059427] Lustre: Skipped 2 previous similar messages [11950.578507] LustreError: 382177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.611692] LustreError: 382177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11961.312344] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11961.652257] Lustre: server umount lustre-OST0000 complete [11965.933462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.962527] Lustre: Skipped 2 previous similar messages [11965.966367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11965.969285] Lustre: Skipped 6 previous similar messages [11980.256148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11980.538032] Lustre: server umount lustre-MDT0000 complete [11984.047678] LustreError: 380335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046781 with bad export cookie 12181113124001187850 [11984.067278] LustreError: 380335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11984.067875] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11990.468657] Lustre: server umount lustre-MDT0001 complete [12001.741979] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12004.581716] Key type lgssc unregistered [12004.903893] LNet: 385188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12004.917676] LNetError: 385188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12004.935835] LNet: Removed LNI 192.168.201.130@tcp [12005.757636] Key type .llcrypt unregistered [12005.765145] Key type ._llcrypt unregistered [12057.670685] Key type ._llcrypt registered [12057.676920] Key type .llcrypt registered [12057.795969] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12059.002448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12059.135623] alg: No test for adler32 (adler32-zlib) [12060.178772] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [12060.424880] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12062.073948] Key type lgssc registered [12063.043926] Lustre: Echo OBD driver; http://www.lustre.org/ [12069.495445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12069.512841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.734866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12070.754435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12070.963378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12075.112469] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12084.509091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.665602] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12084.692967] Lustre: 387990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12085.117951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12090.635726] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12100.570605] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12107.528734] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12110.818520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12110.825926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12110.845000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.893089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.895634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.912777] Lustre: Skipped 1 previous similar message [12116.869858] Lustre: server umount lustre-MDT0000 complete [12120.820399] LustreError: 387982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786046917 with bad export cookie 8213729807580837131 [12120.830691] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12120.833302] LustreError: 387982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12121.088598] Lustre: server umount lustre-MDT0001 complete [12131.379611] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:08:47 (1786046927) [12139.295602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.011430] LustreError: 390502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.027638] LustreError: 390502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12140.119206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12145.122964] LustreError: 390503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.707567] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12150.242430] LustreError: 390502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.370893] LustreError: 390503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.146054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.488297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12160.606842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12165.008593] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12172.932771] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12179.123224] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.117601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12186.302630] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12186.571738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12193.469880] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12201.155796] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12209.043583] Lustre: Permanently deactivating lustre-OST0000 [12209.064107] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12217.890552] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12220.619444] LustreError: 393540:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12220.628642] LustreError: 393540:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12220.640488] LustreError: 393540:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12224.337709] LustreError: 393589:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12224.341836] LustreError: 393589:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12232.795968] Lustre: server umount lustre-OST0000 complete [12235.906332] LustreError: 393839:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12235.911507] LustreError: 393839:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12240.865948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12240.887833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12240.894156] Lustre: Skipped 1 previous similar message [12245.985958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.997716] Lustre: Skipped 1 previous similar message [12247.113429] Lustre: server umount lustre-MDT0000 complete [12250.972083] LustreError: 390483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047048 with bad export cookie 8213729807580837712 [12250.981427] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.985344] LustreError: 390483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12261.604790] Lustre: server umount lustre-MDT0001 complete [12273.943211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.851252] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12283.476327] LustreError: 395000:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12290.412305] Lustre: 395339:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.130@tcp [12292.694470] Lustre: server umount MGS complete [12304.935534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12305.356481] LustreError: 396062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12305.376405] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12305.381264] Lustre: Skipped 1 previous similar message [12305.434522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12310.331839] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12315.618303] LustreError: 396063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12315.629571] LustreError: 396063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12317.979286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.241604] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12323.159574] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12330.340962] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12337.409193] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12339.970654] Lustre: Permanently reactivating lustre-OST0000 [12340.017423] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12340.022550] Lustre: Skipped 2 previous similar messages [12366.816468] LustreError: 397798:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12374.944928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12375.362359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12375.367348] Lustre: Skipped 1 previous similar message [12382.653767] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12391.813994] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12399.914788] Lustre: server umount lustre-OST0000 complete [12405.229761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12405.245882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.264669] Lustre: Skipped 1 previous similar message [12405.275697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12414.541244] Lustre: server umount lustre-MDT0000 complete [12417.513102] LustreError: 397303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.546598] LustreError: 397303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12418.528241] Lustre: 387313:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786047199/real 1786047199] req@00000000a263969d x1872805869226368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786047215 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12418.562317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.585735] Lustre: Skipped 1 previous similar message [12419.893834] LustreError: 396044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047216 with bad export cookie 8213729807580838762 [12419.903053] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.907568] LustreError: 396044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12420.530157] Lustre: server umount lustre-MDT0001 complete [12434.390330] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12437.608680] Key type lgssc unregistered [12437.843549] LNet: 400431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12437.858953] LNetError: 400431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12437.904505] LNet: Removed LNI 192.168.201.130@tcp [12438.988277] Key type .llcrypt unregistered [12438.996465] Key type ._llcrypt unregistered [12458.942804] Key type ._llcrypt registered [12458.945922] Key type .llcrypt registered [12459.115528] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [12476.768466] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12477.902491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12478.132240] alg: No test for adler32 (adler32-zlib) [12479.447046] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [12479.678838] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12481.384181] Key type lgssc registered [12482.597934] Lustre: Echo OBD driver; http://www.lustre.org/ [12524.223653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12524.240062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.570066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12525.673996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12525.767491] Lustre: lustre-MDT0000: new disk, initializing [12525.866971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12525.898637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12529.996236] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12539.793840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.919838] Lustre: 403589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12539.994569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12539.998406] Lustre: Skipped 1 previous similar message [12540.048237] Lustre: lustre-MDT0001: new disk, initializing [12540.120952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12540.163165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12540.180548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12544.495951] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12554.466833] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12560.012552] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12566.200573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12566.566762] Lustre: lustre-OST0000: new disk, initializing [12566.570201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12566.581733] Lustre: 405557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12566.657976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12572.159435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12575.248824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12575.254534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12575.379628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12580.908352] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12585.282541] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12585.468229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12591.049821] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12591.327858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12595.689513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12595.694109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.722350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12596.193744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12596.204062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12596.223118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12596.232585] Lustre: Skipped 1 previous similar message [12599.900459] Lustre: server umount lustre-OST0000 complete [12606.434680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12606.460365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12606.462281] Lustre: Skipped 1 previous similar message [12606.481058] Lustre: Skipped 1 previous similar message [12610.522050] Lustre: server umount lustre-MDT0000 complete [12614.267111] LustreError: 403582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047411 with bad export cookie 4688489302679198715 [12614.273522] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12614.282588] LustreError: 403582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12614.900521] Lustre: server umount lustre-MDT0001 complete [12624.486431] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12626.332612] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:17:01 (1786047421) [12633.487632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12634.088427] LustreError: 408017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12634.110355] LustreError: 408017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12634.206761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12638.550489] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12639.205537] LustreError: 408018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.324741] LustreError: 408017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12647.430447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.981390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12653.929936] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12661.992849] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12668.320570] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12675.940253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12676.402856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12682.190976] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12689.048804] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12690.789645] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12693.162732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12698.086020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12698.096144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12698.109563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12698.115657] Lustre: Skipped 1 previous similar message [12703.203301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12703.211797] Lustre: Skipped 1 previous similar message [12703.952368] Lustre: server umount lustre-OST0000 complete [12708.321899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12708.338359] Lustre: Skipped 2 previous similar messages [12713.442482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12713.450063] Lustre: Skipped 3 previous similar messages [12714.014879] Lustre: server umount lustre-MDT0000 complete [12717.651840] LustreError: 407999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047514 with bad export cookie 4688489302679199597 [12717.656389] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12717.669365] LustreError: 407999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12718.129511] Lustre: server umount lustre-MDT0001 complete [12731.308692] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12734.195806] Key type lgssc unregistered [12734.491069] LNet: 411746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12734.505107] LNetError: 411746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12734.530346] LNet: Removed LNI 192.168.201.130@tcp [12735.335142] Key type .llcrypt unregistered [12735.338480] Key type ._llcrypt unregistered [12760.717396] Key type ._llcrypt registered [12760.719267] Key type .llcrypt registered [12760.802492] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12761.001566] Key type .llcrypt unregistered [12761.002640] Key type ._llcrypt unregistered [12780.368534] Key type ._llcrypt registered [12780.370600] Key type .llcrypt registered [12780.441798] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12781.455768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12781.495917] alg: No test for adler32 (adler32-zlib) [12782.760199] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [12783.148802] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12784.800259] Key type lgssc registered [12785.914678] Lustre: Echo OBD driver; http://www.lustre.org/ [12793.353588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12793.379825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12794.892255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12799.987918] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12804.067899] LustreError: 413966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12804.098620] LustreError: 413966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12808.418074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12808.797662] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.833337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12814.306226] LustreError: 413966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.338298] LustreError: 413966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12814.613211] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12819.426112] LustreError: 413963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12819.449139] LustreError: 413963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12819.461773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12823.170595] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12824.551561] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12824.574573] LustreError: 413967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12830.272420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12830.741947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12835.541596] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12845.027904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12845.031894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.039093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12846.052724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.055893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12846.071726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12846.085657] Lustre: Skipped 1 previous similar message [12846.846981] Lustre: server umount lustre-OST0000 complete [12851.170282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12851.177091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12851.188269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12851.196820] Lustre: Skipped 1 previous similar message [12853.455199] Lustre: server umount lustre-MDT0000 complete [12856.289301] LustreError: 413961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.305953] LustreError: 413961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12857.747070] LustreError: 415709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047654 with bad export cookie 10422604685452422519 [12857.750199] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12857.757895] LustreError: 415709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12858.189562] Lustre: server umount lustre-MDT0001 complete [12871.857671] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [12874.504934] Key type lgssc unregistered [12874.880510] LNet: 417117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12874.887017] LNetError: 417117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12874.920838] LNet: Removed LNI 192.168.201.130@tcp [12875.793202] Key type .llcrypt unregistered [12875.795121] Key type ._llcrypt unregistered [12902.759741] Key type ._llcrypt registered [12902.762820] Key type .llcrypt registered [12902.913430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [12904.265745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12904.372666] alg: No test for adler32 (adler32-zlib) [12905.676815] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [12906.086489] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [12907.848194] Key type lgssc registered [12909.829745] Lustre: Echo OBD driver; http://www.lustre.org/ [12919.594224] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12919.615926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12935.200484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12940.325068] LustreError: 418515:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [12967.008446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12973.037575] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12981.452416] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12989.607071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.771466] LustreError: 417852:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c1b2a34 x1872806756223232/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12990.231202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12990.335057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12995.533515] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [12995.554620] LustreError: 418540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12995.584694] LustreError: 418540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13000.680845] LustreError: 418541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.705024] LustreError: 419553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.735574] LustreError: 419553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13005.021342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13005.351944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13010.202341] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13018.490307] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13024.327415] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13036.006499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13036.013708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.015143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13036.093163] Lustre: Skipped 1 previous similar message [13039.326288] Lustre: server umount lustre-OST0000 complete [13046.244116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13046.262855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13046.271346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.271354] Lustre: Skipped 1 previous similar message [13046.292085] Lustre: Skipped 1 previous similar message [13049.463749] Lustre: server umount lustre-MDT0000 complete [13051.368548] LustreError: 419556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.399724] LustreError: 419556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13054.179634] LustreError: 418521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047851 with bad export cookie 13809340550318548325 [13054.187146] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13054.202432] LustreError: 418521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13054.720309] Lustre: server umount lustre-MDT0001 complete [13066.294851] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13068.702330] Key type lgssc unregistered [13069.123348] LNet: 422088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13069.133943] LNetError: 422088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13070.191179] LNet: Removed LNI 192.168.201.130@tcp [13070.850048] Key type .llcrypt unregistered [13070.851492] Key type ._llcrypt unregistered [13093.730910] Key type ._llcrypt registered [13093.733199] Key type .llcrypt registered [13093.849273] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13094.647162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13094.725098] alg: No test for adler32 (adler32-zlib) [13095.736439] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [13095.941882] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13097.600172] Key type lgssc registered [13098.499524] Lustre: Echo OBD driver; http://www.lustre.org/ [13105.871482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13105.883519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.368562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13112.322737] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13117.922703] LustreError: 423504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13117.952803] LustreError: 423504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13123.045980] LustreError: 423505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13123.345459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13123.937014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13129.967712] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13138.446574] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13145.446153] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13151.851734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13152.202684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13153.274906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13159.146181] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13167.642505] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13173.734086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.736932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13173.762071] Lustre: Skipped 1 previous similar message [13173.785704] Lustre: Skipped 2 previous similar messages [13175.354696] Lustre: server umount lustre-MDT0000 complete [13176.034790] LustreError: 423500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.823402] LustreError: 423484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.30@tcp arrived at 1786047974 with bad export cookie 5308488530020879556 [13177.837967] LustreError: 423484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13178.855258] LustreError: 423505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.876687] LustreError: 423505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13179.670144] LustreError: 423483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047976 with bad export cookie 5308488530020878660 [13179.675130] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13179.682831] LustreError: 423483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13183.970125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13183.975056] LustreError: 425993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.992859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.992868] Lustre: Skipped 1 previous similar message [13183.994966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13184.060044] LustreError: 425993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13185.907963] Lustre: server umount lustre-OST0000 complete [13192.503459] Lustre: server umount lustre-MDT0001 complete [13202.249832] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13204.787690] Key type lgssc unregistered [13205.034776] LNet: 427190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13205.041248] LNetError: 427190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13205.057339] LNet: Removed LNI 192.168.201.130@tcp [13205.809128] Key type .llcrypt unregistered [13205.811095] Key type ._llcrypt unregistered [13239.512887] Key type ._llcrypt registered [13239.516599] Key type .llcrypt registered [13239.578955] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13240.632754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13240.838716] alg: No test for adler32 (adler32-zlib) [13241.874763] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [13242.139799] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13243.812414] Key type lgssc registered [13244.700473] Lustre: Echo OBD driver; http://www.lustre.org/ [13253.241169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13253.262254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13254.722925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13259.731611] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13264.866908] LustreError: 428612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.884872] LustreError: 428612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13268.674458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13269.171743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13273.858410] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13282.540687] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13289.826294] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13298.348188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13298.656103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13299.711913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13305.574995] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13313.453443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13319.138426] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13319.171870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13319.177966] Lustre: Skipped 1 previous similar message [13320.466236] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13324.062678] Lustre: server umount lustre-MDT0001 complete [13324.259934] LustreError: 430453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.526326] LustreError: 428607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.549966] LustreError: 428607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13329.383953] LustreError: 430507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.422239] LustreError: 430507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13334.499430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13334.501568] LustreError: 428612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13334.513028] Lustre: Skipped 1 previous similar message [13334.514873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13336.763292] Lustre: server umount lustre-OST0000 complete [13340.999858] Lustre: server umount lustre-MDT0000 complete [13357.229687] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13360.270242] Key type lgssc unregistered [13360.642153] LNet: 432299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13360.649099] LNetError: 432299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13360.660381] LNet: Removed LNI 192.168.201.130@tcp [13361.961777] Key type .llcrypt unregistered [13361.964434] Key type ._llcrypt unregistered [13390.825687] Key type ._llcrypt registered [13390.834028] Key type .llcrypt registered [13390.954272] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [13410.337700] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13411.959373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13412.253547] alg: No test for adler32 (adler32-zlib) [13414.005759] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [13414.365280] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13416.066112] Key type lgssc registered [13417.816929] Lustre: Echo OBD driver; http://www.lustre.org/ [13470.030504] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13490.231493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13490.258310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13491.493919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13491.511664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13491.566273] Lustre: lustre-MDT0000: new disk, initializing [13491.623587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13491.639661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13496.167588] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13516.555692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13516.987322] Lustre: lustre-OST0000: new disk, initializing [13516.999041] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13517.012796] Lustre: Skipped 1 previous similar message [13517.023832] Lustre: 437360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13517.201395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13523.578370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13523.609317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13523.653874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13526.206970] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13536.381334] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13554.265099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13554.395347] Lustre: 436342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13554.454743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13554.502093] Lustre: lustre-MDT0001: new disk, initializing [13554.591582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13554.625141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13554.637287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13559.884595] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13570.034466] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13570.231125] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13575.182083] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13575.404530] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13581.800083] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13581.801274] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13581.821914] Lustre: Skipped 1 previous similar message [13581.824672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13584.638390] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13584.647226] Lustre: Skipped 1 previous similar message [13586.919055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13586.926632] Lustre: Skipped 1 previous similar message [13589.700098] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13594.839643] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.30@tcp (stopping) [13594.844673] Lustre: Skipped 2 previous similar messages [13595.104142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13595.533099] Lustre: server umount lustre-MDT0001 complete [13602.289185] LustreError: 436352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13602.323048] LustreError: 436352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13607.393807] LustreError: 438013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13607.426314] LustreError: 438013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13612.522533] LustreError: 439214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13612.564704] LustreError: 439214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13617.635229] LustreError: 436353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13617.637069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13617.679447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13622.832399] LustreError: 437345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13623.095430] Lustre: server umount lustre-OST0000 complete [13634.712404] Lustre: server umount lustre-MDT0000 complete [13651.464647] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13654.358853] Key type lgssc unregistered [13654.697171] LNet: 441114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13654.701546] LNetError: 441114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13654.715990] LNet: Removed LNI 192.168.201.130@tcp [13655.737528] Key type .llcrypt unregistered [13655.739632] Key type ._llcrypt unregistered [13702.031583] Key type ._llcrypt registered [13702.032906] Key type .llcrypt registered [13702.160277] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [13702.501725] Key type .llcrypt unregistered [13702.505671] Key type ._llcrypt unregistered [13719.764650] Key type ._llcrypt registered [13719.766344] Key type .llcrypt registered [13719.857664] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [13721.116176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13721.290938] alg: No test for adler32 (adler32-zlib) [13722.348446] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [13722.563911] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [13724.232184] Key type lgssc registered [13725.530814] Lustre: Echo OBD driver; http://www.lustre.org/ [13753.813100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13753.863293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.141884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13755.190160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13755.276798] Lustre: lustre-MDT0000: new disk, initializing [13755.388217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13755.407786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13759.884036] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13772.477560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13772.662479] Lustre: 444496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13772.743554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13772.749293] Lustre: Skipped 1 previous similar message [13772.877449] Lustre: lustre-MDT0001: new disk, initializing [13772.976315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13773.035275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13773.049494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13778.185284] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13788.573839] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13795.365842] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13801.948722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13802.234537] Lustre: lustre-OST0000: new disk, initializing [13802.237941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13802.243390] Lustre: 446466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13802.328190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13808.230330] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13808.746917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13808.759426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13808.908322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13818.593588] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13829.602689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13829.606972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13829.624160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13829.627439] Lustre: Skipped 2 previous similar messages [13829.656534] Lustre: Skipped 2 previous similar messages [13835.559589] Lustre: server umount lustre-MDT0000 complete [13839.283883] LustreError: 444490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786048636 with bad export cookie 14948359738761601274 [13839.291378] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13839.309373] LustreError: 444490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13839.724712] Lustre: server umount lustre-MDT0001 complete [13854.633516] Lustre: server umount lustre-OST0000 complete [13867.721906] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:37:43 (1786048663) [13875.463471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.976718] LustreError: 448492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13875.998613] LustreError: 448492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13876.031700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13880.362211] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13881.328764] LustreError: 448493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13881.354746] LustreError: 448493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13886.433599] LustreError: 448492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13889.443307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13889.945296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13894.735914] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13903.199962] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13910.531972] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13932.268323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13932.470399] Lustre: Found index 0 for lustre-OST0000, updating log [13932.631246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13938.700454] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [13947.680401] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13948.017910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13954.263731] Lustre: server umount lustre-OST0000 complete [13971.232189] Lustre: 443029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786048752/real 1786048752] req@000000005f16d2f5 x1872807611933184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786048768 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13971.277489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13971.337566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13973.408442] Lustre: 443030:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786048754/real 1786048754] req@00000000a87e2786 x1872807611934208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786048770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13976.548954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.563666] Lustre: Skipped 1 previous similar message [13978.529445] Lustre: 443028:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786048759/real 1786048759] req@00000000033e7101 x1872807611935488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786048775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13980.904774] Lustre: server umount lustre-MDT0000 complete [13981.665543] LustreError: 448487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13981.692967] LustreError: 448487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13983.726206] Lustre: 443027:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786048764/real 1786048764] req@00000000e816f103 x1872807611936640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786048780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13985.886669] LustreError: 448473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786048782 with bad export cookie 14948359738761602667 [13985.900610] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13985.905283] LustreError: 448473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13986.279973] Lustre: server umount lustre-MDT0001 complete [13998.939587] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:39:53 (1786048793) [13999.073364] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [14630.688038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14631.406639] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14631.438642] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14631.558858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14637.031768] LustreError: 454906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14637.219444] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14641.148341] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14647.271578] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14647.293628] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14649.135225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14649.601567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14654.646809] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14663.910443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14670.673493] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14690.338611] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [14694.330155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14694.738239] Lustre: Found index 0 for lustre-OST0000, updating log [14695.333987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14704.032688] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [14714.946462] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14719.162846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14719.178696] Lustre: Skipped 2 previous similar messages [14719.202737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14724.414259] Lustre: server umount lustre-OST0000 complete [14729.762145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14729.791095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14729.796277] Lustre: Skipped 1 previous similar message [14730.869590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14730.880040] Lustre: Skipped 1 previous similar message [14732.897552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14732.899652] Lustre: Skipped 2 previous similar messages [14735.235901] Lustre: server umount lustre-MDT0000 complete [14735.978589] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14736.001072] LustreError: 454905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14740.524226] LustreError: 454886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786049537 with bad export cookie 14948359738761603752 [14740.532714] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14740.543418] LustreError: 454886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14740.961581] Lustre: server umount lustre-MDT0001 complete [14750.754959] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp [14774.617075] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [14788.350585] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14832.023133] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:53:47 (1786049627) [14854.507806] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [14858.599615] Key type lgssc unregistered [14858.911733] LNet: 461526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14858.920789] LNetError: 461526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14956.199422] LNet: Removed LNI 192.168.201.130@tcp [14957.140505] Key type .llcrypt unregistered [14957.142478] Key type ._llcrypt unregistered [14975.441844] Key type ._llcrypt registered [14975.450081] Key type .llcrypt registered [14975.583667] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [14976.944869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14977.105101] alg: No test for adler32 (adler32-zlib) [14978.255348] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [14978.514747] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [14980.248219] Key type lgssc registered [14981.531395] Lustre: Echo OBD driver; http://www.lustre.org/ [14985.584260] LNet: Added LNI 192.168.201.130@tcp100 [8/256/0/180] [14986.473257] LNet: Added LNI 192.168.201.130@tcp101 [8/256/0/180] [14987.315956] LNet: Added LNI 192.168.201.130@tcp102 [8/256/0/180] [14988.250360] LNet: Added LNI 192.168.201.130@tcp103 [8/256/0/180] [14989.081329] LNet: Added LNI 192.168.201.130@tcp104 [8/256/0/180] [14989.829710] LNet: Added LNI 192.168.201.130@tcp105 [8/256/0/180] [14990.542594] LNet: Added LNI 192.168.201.130@tcp106 [8/256/0/180] [14991.422624] LNet: Added LNI 192.168.201.130@tcp107 [8/256/0/180] [14992.374775] LNet: Added LNI 192.168.201.130@tcp108 [8/256/0/180] [14993.303541] LNet: Added LNI 192.168.201.130@tcp109 [8/256/0/180] [14994.013175] LNet: Added LNI 192.168.201.130@tcp110 [8/256/0/180] [14994.875083] LNet: Added LNI 192.168.201.130@tcp111 [8/256/0/180] [14995.848455] LNet: Added LNI 192.168.201.130@tcp112 [8/256/0/180] [14996.752477] LNet: Added LNI 192.168.201.130@tcp113 [8/256/0/180] [14997.626824] LNet: Added LNI 192.168.201.130@tcp114 [8/256/0/180] [14998.608351] LNet: Added LNI 192.168.201.130@tcp115 [8/256/0/180] [14999.389981] LNet: Added LNI 192.168.201.130@tcp116 [8/256/0/180] [15000.148706] LNet: Added LNI 192.168.201.130@tcp117 [8/256/0/180] [15000.954660] LNet: Added LNI 192.168.201.130@tcp118 [8/256/0/180] [15001.817636] LNet: Added LNI 192.168.201.130@tcp119 [8/256/0/180] [15002.565759] LNet: Added LNI 192.168.201.130@tcp120 [8/256/0/180] [15003.464355] LNet: Added LNI 192.168.201.130@tcp121 [8/256/0/180] [15004.378408] LNet: Added LNI 192.168.201.130@tcp122 [8/256/0/180] [15005.396314] LNet: Added LNI 192.168.201.130@tcp123 [8/256/0/180] [15006.008848] LNet: Added LNI 192.168.201.130@tcp124 [8/256/0/180] [15007.109398] LNet: Added LNI 192.168.201.130@tcp125 [8/256/0/180] [15008.047987] LNet: Added LNI 192.168.201.130@tcp126 [8/256/0/180] [15008.955845] LNet: Added LNI 192.168.201.130@tcp127 [8/256/0/180] [15009.964181] LNet: Added LNI 192.168.201.130@tcp128 [8/256/0/180] [15011.112042] LNet: Added LNI 192.168.201.130@tcp129 [8/256/0/180] [15011.944409] LNet: Added LNI 192.168.201.130@tcp130 [8/256/0/180] [15012.884085] LNet: Added LNI 192.168.201.130@tcp131 [8/256/0/180] [15013.772886] LNet: Added LNI 192.168.201.130@tcp132 [8/256/0/180] [15014.820107] LNet: Added LNI 192.168.201.130@tcp133 [8/256/0/180] [15015.975558] LNet: Added LNI 192.168.201.130@tcp134 [8/256/0/180] [15016.927786] LNet: Added LNI 192.168.201.130@tcp135 [8/256/0/180] [15023.764634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15023.793488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15024.924811] Lustre: 464556:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [15025.034788] LNetError: 464584:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15025.053596] LNetError: Protocol error connecting to 192.168.201.130@tcp118 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15025.068716] LNetError: Skipped 7 previous similar messages [15025.227345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15025.267845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15025.373872] Lustre: lustre-MDT0000: new disk, initializing [15025.456474] Lustre: 464612:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [15025.468233] Lustre: 464612:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15025.488323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15025.527483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15025.654481] LNet: 9 local NIs in recovery (showing 9): 192.168.201.130@tcp122, 192.168.201.130@tcp123, 192.168.201.130@tcp124, 192.168.201.130@tcp125, 192.168.201.130@tcp127, 192.168.201.130@tcp128, 192.168.201.130@tcp118, 192.168.201.130@tcp126, 192.168.201.130@tcp129 [15026.721993] LNetError: 464580:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15026.755377] LNetError: 464580:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [15026.767861] LNetError: Protocol error connecting to 192.168.201.130@tcp123 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15026.782512] LNetError: Skipped 7 previous similar messages [15028.780405] LNetError: 462016:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15028.799194] LNetError: 462016:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15028.809913] LNetError: Protocol error connecting to 192.168.201.130@tcp123 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15031.195978] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15032.866438] LNetError: 464577:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15032.891753] LNetError: Protocol error connecting to 192.168.201.130@tcp123 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15040.033437] LNetError: 464584:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15040.054706] LNetError: Protocol error connecting to 192.168.201.130@tcp123 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15043.330071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15043.560327] Lustre: 464571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15043.690251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15043.695266] Lustre: Skipped 1 previous similar message [15043.850081] Lustre: lustre-MDT0001: new disk, initializing [15043.949301] Lustre: 465376:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [15043.970277] Lustre: 465376:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15044.139034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15044.209674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15044.231972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15050.195394] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15051.352592] Lustre: 465693:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [15051.367622] Lustre: 465693:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15056.354050] LNetError: 462015:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15056.374416] LNetError: Protocol error connecting to 192.168.201.130@tcp120 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15064.112659] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15071.153676] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15073.774455] LNetError: 464577:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15073.790480] LNetError: 464577:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [15073.802921] LNetError: Protocol error connecting to 192.168.201.130@tcp126 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15073.820843] LNetError: Skipped 7 previous similar messages [15078.740609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15078.937730] Lustre: lustre-OST0000: new disk, initializing [15078.940595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15078.945602] Lustre: 466549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15079.011979] Lustre: 466554:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.130@tcp: can't add NID 192.168.201.130@tcp130: rc = -75 [15079.015849] Lustre: 466554:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15079.073478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15085.091979] LNet: 1 local NIs in recovery (showing 1): 192.168.201.130@tcp123 [15085.446732] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15086.194024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15086.205969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15086.428861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15095.259413] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15106.534559] LNetError: 464582:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.130 [15106.547965] LNetError: 464582:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [15106.565287] LNetError: Protocol error connecting to 192.168.201.130@tcp117 at host 192.168.201.130:988: is it running a compatible version of Lustre? [15106.581707] LNetError: Skipped 16 previous similar messages [15107.054971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15107.060987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15107.062699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15107.068892] Lustre: Skipped 1 previous similar message [15107.096333] Lustre: Skipped 1 previous similar message [15109.790808] Lustre: server umount lustre-OST0000 complete [15115.748345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15115.766467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15115.789229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15117.285593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15117.287964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15120.476051] Lustre: server umount lustre-MDT0000 complete [15124.052095] LustreError: 464565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786049921 with bad export cookie 12053565165605584237 [15124.073403] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15124.333656] Lustre: server umount lustre-MDT0001 complete [15136.419264] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15139.100555] Key type lgssc unregistered [15139.512644] LNet: 468464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15139.523856] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15140.576195] LNet: Removed LNI 192.168.201.130@tcp [15140.579570] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15140.594475] LNet: Removed LNI 192.168.201.130@tcp100 [15141.600166] LNet: Removed LNI 192.168.201.130@tcp101 [15141.602220] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15141.613339] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15141.620555] LNet: Removed LNI 192.168.201.130@tcp102 [15142.689954] LNet: Removed LNI 192.168.201.130@tcp103 [15143.715369] LNet: Removed LNI 192.168.201.130@tcp104 [15143.718638] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15143.737283] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15143.759310] LNet: Removed LNI 192.168.201.130@tcp105 [15144.800176] LNet: Removed LNI 192.168.201.130@tcp106 [15145.826372] LNet: Removed LNI 192.168.201.130@tcp107 [15146.912215] LNet: Removed LNI 192.168.201.130@tcp108 [15147.936179] LNet: Removed LNI 192.168.201.130@tcp109 [15147.938280] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15147.943653] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15147.948443] LNet: Removed LNI 192.168.201.130@tcp110 [15148.965115] LNet: Removed LNI 192.168.201.130@tcp111 [15149.984173] LNet: Removed LNI 192.168.201.130@tcp112 [15151.008199] LNet: Removed LNI 192.168.201.130@tcp113 [15152.032185] LNet: Removed LNI 192.168.201.130@tcp114 [15153.056611] LNet: Removed LNI 192.168.201.130@tcp115 [15154.080169] LNet: Removed LNI 192.168.201.130@tcp116 [15155.105434] LNet: Removed LNI 192.168.201.130@tcp117 [15156.128163] LNet: Removed LNI 192.168.201.130@tcp118 [15156.133789] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15156.140981] LNetError: 468464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15156.163286] LNet: Removed LNI 192.168.201.130@tcp119 [15157.216739] LNet: Removed LNI 192.168.201.130@tcp120 [15158.240734] LNet: Removed LNI 192.168.201.130@tcp121 [15159.264227] LNet: Removed LNI 192.168.201.130@tcp122 [15159.267286] LNet: Removed LNI 192.168.201.130@tcp123 [15160.291096] LNet: Removed LNI 192.168.201.130@tcp124 [15161.312325] LNet: Removed LNI 192.168.201.130@tcp125 [15162.336845] LNet: Removed LNI 192.168.201.130@tcp126 [15163.360169] LNet: Removed LNI 192.168.201.130@tcp127 [15164.385389] LNet: Removed LNI 192.168.201.130@tcp128 [15165.408321] LNet: Removed LNI 192.168.201.130@tcp129 [15165.412195] LNet: Removed LNI 192.168.201.130@tcp130 [15165.420901] LNet: Removed LNI 192.168.201.130@tcp131 [15165.426954] LNet: Removed LNI 192.168.201.130@tcp132 [15165.433968] LNet: Removed LNI 192.168.201.130@tcp133 [15165.439983] LNet: Removed LNI 192.168.201.130@tcp134 [15165.457357] LNet: Removed LNI 192.168.201.130@tcp135 [15166.383222] Key type .llcrypt unregistered [15166.390332] Key type ._llcrypt unregistered [15194.761366] Key type ._llcrypt registered [15194.765385] Key type .llcrypt registered [15194.914927] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [15213.333943] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15214.991962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15215.047487] alg: No test for adler32 (adler32-zlib) [15216.086488] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [15216.320174] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15217.977545] Key type lgssc registered [15219.585461] Lustre: Echo OBD driver; http://www.lustre.org/ [15259.721503] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:00:55 (1786050055) [15280.603777] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15283.391694] Key type lgssc unregistered [15283.689040] LNet: 472574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15283.708466] LNetError: 472574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15283.732787] LNet: Removed LNI 192.168.201.130@tcp [15284.578738] Key type .llcrypt unregistered [15284.581418] Key type ._llcrypt unregistered [15302.558236] Key type ._llcrypt registered [15302.565694] Key type .llcrypt registered [15302.690679] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15303.979708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15304.003031] alg: No test for adler32 (adler32-zlib) [15305.306482] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [15305.495849] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15307.168211] Key type lgssc registered [15308.429859] Lustre: Echo OBD driver; http://www.lustre.org/ [15318.063621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15318.097836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15319.404553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15319.450575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15319.558938] Lustre: lustre-MDT0000: new disk, initializing [15319.677635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15319.713238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15324.083726] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15334.994061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15335.150325] Lustre: 473789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15335.219623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15335.225208] Lustre: Skipped 1 previous similar message [15335.352153] Lustre: lustre-MDT0001: new disk, initializing [15335.479321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15335.558835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15335.572578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15340.469935] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15350.840190] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15357.639392] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15365.891361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15366.198500] Lustre: lustre-OST0000: new disk, initializing [15366.202711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15366.214880] Lustre: 475756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15366.320773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15372.430708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15372.453339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15372.626209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15373.379184] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15384.955459] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15390.322578] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing lnet_if_list [15397.736078] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15397.781541] LNet: Added LNI 192.168.201.130@tcp10 [8/256/0/180] [15403.622829] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15403.658867] LNet: Added LNI 192.168.201.130@tcp11 [8/256/0/180] [15407.999800] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15408.050683] LNet: Added LNI 192.168.201.130@tcp12 [8/256/0/180] [15413.132264] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15413.185367] LNet: Added LNI 192.168.201.130@tcp13 [8/256/0/180] [15417.975805] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15418.016352] LNet: Added LNI 192.168.201.130@tcp14 [8/256/0/180] [15423.316319] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15423.364397] LNet: Added LNI 192.168.201.130@tcp15 [8/256/0/180] [15428.679254] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15428.710911] LNet: Added LNI 192.168.201.130@tcp16 [8/256/0/180] [15434.105943] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15434.161685] LNet: Added LNI 192.168.201.130@tcp17 [8/256/0/180] [15440.164879] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15440.209692] LNet: Added LNI 192.168.201.130@tcp18 [8/256/0/180] [15446.986206] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15447.029819] LNet: Added LNI 192.168.201.130@tcp19 [8/256/0/180] [15454.545329] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15454.582710] LNet: Added LNI 192.168.201.130@tcp20 [8/256/0/180] [15461.181392] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15461.206370] LNet: Added LNI 192.168.201.130@tcp21 [8/256/0/180] [15466.590931] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15466.617214] LNet: Added LNI 192.168.201.130@tcp22 [8/256/0/180] [15472.281644] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15472.322888] LNet: Added LNI 192.168.201.130@tcp23 [8/256/0/180] [15478.319562] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15478.348374] LNet: Added LNI 192.168.201.130@tcp24 [8/256/0/180] [15483.447882] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15483.478304] LNet: Added LNI 192.168.201.130@tcp25 [8/256/0/180] [15490.305655] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15490.340850] LNet: Added LNI 192.168.201.130@tcp26 [8/256/0/180] [15495.725837] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15495.751219] LNet: Added LNI 192.168.201.130@tcp27 [8/256/0/180] [15500.988358] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15501.007035] LNet: Added LNI 192.168.201.130@tcp28 [8/256/0/180] [15507.420752] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15507.446885] LNet: Added LNI 192.168.201.130@tcp29 [8/256/0/180] [15512.699401] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15512.730546] LNet: Added LNI 192.168.201.130@tcp30 [8/256/0/180] [15518.641605] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15518.666373] LNet: Added LNI 192.168.201.130@tcp31 [8/256/0/180] [15524.358278] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15524.371834] LNet: Added LNI 192.168.201.130@tcp32 [8/256/0/180] [15530.959730] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15530.993724] LNet: Added LNI 192.168.201.130@tcp33 [8/256/0/180] [15537.374989] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15537.401612] LNet: Added LNI 192.168.201.130@tcp34 [8/256/0/180] [15544.943832] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15544.963926] LNet: Added LNI 192.168.201.130@tcp35 [8/256/0/180] [15552.916967] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15552.965042] LNet: Added LNI 192.168.201.130@tcp36 [8/256/0/180] [15560.048487] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15560.067245] LNet: Added LNI 192.168.201.130@tcp37 [8/256/0/180] [15566.891457] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15566.927639] LNet: Added LNI 192.168.201.130@tcp38 [8/256/0/180] [15574.004612] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15574.033922] LNet: Added LNI 192.168.201.130@tcp39 [8/256/0/180] [15581.006851] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15581.037881] LNet: Added LNI 192.168.201.130@tcp40 [8/256/0/180] [15588.471501] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15588.502967] LNet: Added LNI 192.168.201.130@tcp41 [8/256/0/180] [15597.272526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15597.300596] LNet: Added LNI 192.168.201.130@tcp42 [8/256/0/180] [15605.421137] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15605.442317] LNet: Added LNI 192.168.201.130@tcp43 [8/256/0/180] [15614.176606] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15614.205264] LNet: Added LNI 192.168.201.130@tcp44 [8/256/0/180] [15621.665034] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15621.680078] LNet: Added LNI 192.168.201.130@tcp45 [8/256/0/180] [15628.828685] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15628.856869] LNet: Added LNI 192.168.201.130@tcp46 [8/256/0/180] [15637.758825] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15637.815917] LNet: Added LNI 192.168.201.130@tcp47 [8/256/0/180] [15646.427649] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15646.461032] LNet: Added LNI 192.168.201.130@tcp48 [8/256/0/180] [15656.180467] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15656.212541] LNet: Added LNI 192.168.201.130@tcp49 [8/256/0/180] [15664.767587] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15664.797147] LNet: Added LNI 192.168.201.130@tcp50 [8/256/0/180] [15685.102262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15685.115492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15685.120361] Lustre: Skipped 1 previous similar message [15685.133700] Lustre: Skipped 2 previous similar messages [15690.222257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15690.227036] Lustre: Skipped 1 previous similar message [15690.959471] Lustre: server umount lustre-MDT0000 complete [15695.011385] LustreError: 474582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786050492 with bad export cookie 13728680629291258071 [15695.016527] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15695.330646] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15695.340588] Lustre: Skipped 1 previous similar message [15695.501036] Lustre: server umount lustre-MDT0001 complete [15715.538852] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15724.900346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15740.692211] LustreError: 475750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15740.721879] LustreError: 475750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15740.766819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15745.488551] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15746.048401] LustreError: 475751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15750.186156] LustreError: 475750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15750.188128] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15754.027417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15754.347129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15758.767928] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15760.360082] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15766.141836] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15771.432396] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15792.102280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15792.115435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15792.132025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15795.175201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15795.182832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15795.195341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15798.282346] Lustre: server umount lustre-OST0000 complete [15805.413392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15805.427987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15805.443099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15805.450889] Lustre: Skipped 2 previous similar messages [15808.577598] Lustre: server umount lustre-MDT0000 complete [15811.554738] LustreError: 501442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15811.585966] LustreError: 501442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15812.717592] LustreError: 473781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786050609 with bad export cookie 13728680629291260661 [15812.727099] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15812.730636] LustreError: 473781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15813.239557] Lustre: server umount lustre-MDT0001 complete [15826.653349] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [15829.756940] Key type lgssc unregistered [15830.109294] LNet: 502807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15830.115168] LNetError: 502807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15831.136174] LNet: Removed LNI 192.168.201.130@tcp [15831.140450] LNetError: 502807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15831.146192] LNet: Removed LNI 192.168.201.130@tcp10 [15831.149316] LNet: Removed LNI 192.168.201.130@tcp11 [15831.151540] LNet: Removed LNI 192.168.201.130@tcp12 [15831.157929] LNet: Removed LNI 192.168.201.130@tcp13 [15831.161725] LNet: Removed LNI 192.168.201.130@tcp14 [15831.164312] LNet: Removed LNI 192.168.201.130@tcp15 [15831.174941] LNet: Removed LNI 192.168.201.130@tcp16 [15831.181328] LNet: Removed LNI 192.168.201.130@tcp17 [15831.192360] LNet: Removed LNI 192.168.201.130@tcp18 [15831.200388] LNet: Removed LNI 192.168.201.130@tcp19 [15831.220691] LNet: Removed LNI 192.168.201.130@tcp20 [15831.227393] LNet: Removed LNI 192.168.201.130@tcp21 [15831.231289] LNet: Removed LNI 192.168.201.130@tcp22 [15831.237237] LNet: Removed LNI 192.168.201.130@tcp23 [15831.242921] LNet: Removed LNI 192.168.201.130@tcp24 [15831.247711] LNet: Removed LNI 192.168.201.130@tcp25 [15831.253205] LNet: Removed LNI 192.168.201.130@tcp26 [15831.254764] LNet: Removed LNI 192.168.201.130@tcp27 [15831.256052] LNet: Removed LNI 192.168.201.130@tcp28 [15831.259614] LNet: Removed LNI 192.168.201.130@tcp29 [15831.264678] LNet: Removed LNI 192.168.201.130@tcp30 [15831.269676] LNet: Removed LNI 192.168.201.130@tcp31 [15831.272829] LNet: Removed LNI 192.168.201.130@tcp32 [15831.280527] LNet: Removed LNI 192.168.201.130@tcp33 [15831.283535] LNet: Removed LNI 192.168.201.130@tcp34 [15831.285963] LNet: Removed LNI 192.168.201.130@tcp35 [15831.287883] LNet: Removed LNI 192.168.201.130@tcp36 [15831.289723] LNet: Removed LNI 192.168.201.130@tcp37 [15831.291583] LNet: Removed LNI 192.168.201.130@tcp38 [15831.293906] LNet: Removed LNI 192.168.201.130@tcp39 [15831.296896] LNet: Removed LNI 192.168.201.130@tcp40 [15831.300266] LNet: Removed LNI 192.168.201.130@tcp41 [15831.302512] LNet: Removed LNI 192.168.201.130@tcp42 [15831.305394] LNet: Removed LNI 192.168.201.130@tcp43 [15831.312789] LNet: Removed LNI 192.168.201.130@tcp44 [15831.320990] LNet: Removed LNI 192.168.201.130@tcp45 [15831.329141] LNet: Removed LNI 192.168.201.130@tcp46 [15831.336287] LNet: Removed LNI 192.168.201.130@tcp47 [15831.340396] LNet: Removed LNI 192.168.201.130@tcp48 [15831.344221] LNet: Removed LNI 192.168.201.130@tcp49 [15831.354412] LNet: Removed LNI 192.168.201.130@tcp50 [15832.278181] Key type .llcrypt unregistered [15832.285955] Key type ._llcrypt unregistered [15853.770658] Key type ._llcrypt registered [15853.772629] Key type .llcrypt registered [15853.916750] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [15871.900258] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [15873.252850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15873.330532] alg: No test for adler32 (adler32-zlib) [15874.365779] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [15874.618375] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [15876.272203] Key type lgssc registered [15877.584862] Lustre: Echo OBD driver; http://www.lustre.org/ [15914.087860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15914.128256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15915.445822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15915.468466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15915.550958] Lustre: lustre-MDT0000: new disk, initializing [15915.621428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15915.642511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15920.271384] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15932.377877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15932.523780] Lustre: 506006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15932.590174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15932.592456] Lustre: Skipped 1 previous similar message [15932.669544] Lustre: lustre-MDT0001: new disk, initializing [15932.776411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15932.823397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15932.837299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15937.846045] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15947.845744] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15954.931666] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15962.924609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15963.256620] Lustre: *** cfs_fail_loc=90f, val=0*** [15963.404466] Lustre: lustre-OST0000: new disk, initializing [15963.419970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15963.430589] Lustre: 508021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15963.606159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15970.114251] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [15970.893773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15970.905445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15971.105808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15980.679457] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15991.779957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15991.786383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15991.796510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15994.640988] Lustre: server umount lustre-OST0000 complete [15999.457959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15999.469199] LustreError: Skipped 1 previous similar message [15999.473175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15999.481946] Lustre: Skipped 1 previous similar message [15999.485224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15999.487589] Lustre: Skipped 1 previous similar message [16002.023440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16002.023609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16002.053886] Lustre: Skipped 1 previous similar message [16005.164198] Lustre: server umount lustre-MDT0000 complete [16008.675195] LustreError: 505997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786050805 with bad export cookie 8465044726343676241 [16008.680646] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16008.690159] LustreError: 505997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16009.066452] Lustre: server umount lustre-MDT0001 complete [16023.914409] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [16027.914051] Key type lgssc unregistered [16028.478766] LNet: 509933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16028.498868] LNetError: 509933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16028.533489] LNet: Removed LNI 192.168.201.130@tcp [16029.656461] Key type .llcrypt unregistered [16029.659931] Key type ._llcrypt unregistered [16051.081756] Key type ._llcrypt registered [16051.085601] Key type .llcrypt registered [16051.228178] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16052.408416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16052.445543] alg: No test for adler32 (adler32-zlib) [16053.568545] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [16053.872567] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [16055.552191] Key type lgssc registered [16056.942397] Lustre: Echo OBD driver; http://www.lustre.org/ [16066.179259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16066.194494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16067.893071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16072.951455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16077.284687] LustreError: 511205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16077.311565] LustreError: 511205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16082.413504] LustreError: 511204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16084.190654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16084.529790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16090.204138] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16100.793463] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16107.567684] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16115.015582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16115.373674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16121.935768] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16131.831280] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16149.146832] loop0: detected capacity change from 0 to 204800000 [16149.395904] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16149.404970] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16149.413860] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16156.532285] loop0: detected capacity change from 0 to 204800000 [16156.895696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16156.907618] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16156.927052] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16163.527781] loop0: detected capacity change from 0 to 204800000 [16170.653665] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16170.825835] Lustre: fs15246-OST0000: new disk, initializing [16170.830424] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16170.837492] Lustre: 514812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16170.940409] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16179.274236] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16191.746355] loop1: detected capacity change from 0 to 204800000 [16198.938540] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16199.173636] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16199.308581] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16199.468559] Lustre: fs15246-MDT0000: new disk, initializing [16199.624087] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16199.637876] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16199.733108] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16200.804651] Lustre: 515936:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f20b37e2 x1872810056141952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786051013 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16205.081132] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16209.921820] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16209.929564] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16209.968331] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16231.914196] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16231.941519] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16236.581383] Lustre: server umount fs15246-MDT0000 complete [16237.037709] LustreError: 511198:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16242.179455] LustreError: 511201:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16247.293740] LustreError: 511201:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16248.031397] Lustre: server umount fs15246-OST0000 complete [16262.630593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16262.658382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16267.745290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16267.755654] Lustre: Skipped 2 previous similar messages [16268.251614] Lustre: server umount lustre-OST0000 complete [16272.868776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16272.879783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16272.892558] Lustre: Skipped 2 previous similar messages [16272.902502] Lustre: Skipped 1 previous similar message [16277.989717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16277.997329] Lustre: Skipped 1 previous similar message [16286.180458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16286.543368] Lustre: server umount lustre-MDT0000 complete [16288.226950] LustreError: 511205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16288.244790] LustreError: 511205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16290.670483] LustreError: 511184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051087 with bad export cookie 4267807633374937528 [16290.673075] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16290.677936] LustreError: 511184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16291.100336] Lustre: server umount lustre-MDT0001 complete [16303.345757] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [16306.554258] Key type lgssc unregistered [16306.858528] LNet: 519037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16306.864367] LNetError: 519037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16306.881212] LNet: Removed LNI 192.168.201.130@tcp [16307.637110] Key type .llcrypt unregistered [16307.638791] Key type ._llcrypt unregistered [16328.980777] Key type ._llcrypt registered [16328.985199] Key type .llcrypt registered [16329.152360] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16330.248586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16330.310097] alg: No test for adler32 (adler32-zlib) [16331.542701] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [16331.885277] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [16333.632179] Key type lgssc registered [16334.764480] Lustre: Echo OBD driver; http://www.lustre.org/ [16379.735073] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [16395.064430] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16443.247032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16443.304717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16444.618610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16444.641730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16444.700611] Lustre: lustre-MDT0000: new disk, initializing [16444.834373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16444.851539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16451.816746] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16463.967751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16464.096423] Lustre: 523526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16464.141394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16464.157848] Lustre: Skipped 1 previous similar message [16464.255533] Lustre: lustre-MDT0001: new disk, initializing [16464.351805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16464.388696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16464.402108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16469.872490] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16480.253648] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16487.843845] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16495.607657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16495.965515] Lustre: lustre-OST0000: new disk, initializing [16495.979387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16495.989969] Lustre: 525492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16496.147514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16502.871957] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [16505.921126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16505.939424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16506.131111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16513.152558] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16517.690808] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16517.809702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16522.719608] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16522.919859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16526.822112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16526.822257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16526.851965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16526.868233] Lustre: Skipped 1 previous similar message [16531.023819] Lustre: server umount lustre-OST0000 complete [16536.035065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16536.053267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16536.073699] Lustre: Skipped 1 previous similar message [16536.085216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16537.064538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16537.086150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16537.092086] Lustre: Skipped 1 previous similar message [16541.647517] Lustre: server umount lustre-MDT0000 complete [16546.308310] LustreError: 524320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786051343 with bad export cookie 13049260342611590507 [16546.324579] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16546.753774] Lustre: server umount lustre-MDT0001 complete [16576.018754] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [16589.361666] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [16625.173276] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16300 sec ========= 17:23:40 (1786051420) [16627.489504] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:23:42 (1786051422) === [16631.997993] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:23:47 (1786051427) === [16677.165836] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [16680.765246] Key type lgssc unregistered [16681.175671] LNet: 532144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16681.182862] LNetError: 532144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16681.196786] LNet: Removed LNI 192.168.201.130@tcp [16682.189971] Key type .llcrypt unregistered [16682.192188] Key type ._llcrypt unregistered