[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 543316890 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006497] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009109] LSM: Security Framework initializing [ 0.010043] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012046] *** VALIDATE selinux *** [ 0.020042] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024453] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.029359] *** VALIDATE proc *** [ 0.030240] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.032206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035024] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.039884] debug: unmapping init [mem 0xffffffff86c59000-0xffffffff86c60fff] [ 0.041229] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043027] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.050281] rcu: Hierarchical SRCU implementation. [ 0.052617] smp: Bringing up secondary CPUs ... [ 0.053526] x86: Booting SMP configuration: [ 0.054014] .... node #0, CPUs: #1 #2 #3 [ 0.061678] smp: Brought up 1 node, 4 CPUs [ 0.063008] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.107291] node 0 deferred pages initialised in 41ms [ 0.111006] devtmpfs: initialized [ 0.112230] x86/mm: Memory block size: 128MB [ 0.115873] gcov: version magic: 0x41383552 [ 0.117381] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120518] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123388] pinctrl core: initialized pinctrl subsystem [ 0.124170] [ 0.124692] ************************************************************* [ 0.127009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129009] ** ** [ 0.131014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133010] ** ** [ 0.135009] ** This means that this kernel is built to expose internal ** [ 0.137010] ** IOMMU data structures, which may compromise security on ** [ 0.139010] ** your system. ** [ 0.140049] ** ** [ 0.142008] ** If you see this message and you are not debugging the ** [ 0.144010] ** kernel, report this immediately to your vendor! ** [ 0.146009] ** ** [ 0.147011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149008] ************************************************************* [ 0.150725] NET: Registered protocol family 16 [ 0.152548] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159079] cpuidle: using governor menu [ 0.160796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162425] PCI: Using configuration type 1 for base access [ 0.164176] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178274] cryptd: max_cpu_qlen set to 1000 [ 0.181007] ACPI: Added _OSI(Module Device) [ 0.182011] ACPI: Added _OSI(Processor Device) [ 0.184015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.185017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192383] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199081] ACPI: Interpreter enabled [ 0.201099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204092] ACPI: Using IOAPIC for interrupt routing [ 0.210155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215449] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226927] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238358] acpiphp: Slot [2] registered [ 0.239086] acpiphp: Slot [5] registered [ 0.241108] acpiphp: Slot [6] registered [ 0.243126] acpiphp: Slot [7] registered [ 0.248151] acpiphp: Slot [8] registered [ 0.249087] acpiphp: Slot [9] registered [ 0.250156] acpiphp: Slot [10] registered [ 0.251078] acpiphp: Slot [3] registered [ 0.252000] acpiphp: Slot [4] registered [ 0.252000] acpiphp: Slot [11] registered [ 0.254091] acpiphp: Slot [12] registered [ 0.255078] acpiphp: Slot [13] registered [ 0.256069] acpiphp: Slot [14] registered [ 0.257075] acpiphp: Slot [15] registered [ 0.263122] acpiphp: Slot [16] registered [ 0.272148] acpiphp: Slot [17] registered [ 0.275126] acpiphp: Slot [18] registered [ 0.278146] acpiphp: Slot [19] registered [ 0.280089] acpiphp: Slot [20] registered [ 0.281083] acpiphp: Slot [21] registered [ 0.284159] acpiphp: Slot [22] registered [ 0.286387] acpiphp: Slot [23] registered [ 0.288270] acpiphp: Slot [24] registered [ 0.293081] acpiphp: Slot [25] registered [ 0.298090] acpiphp: Slot [26] registered [ 0.300098] acpiphp: Slot [27] registered [ 0.302102] acpiphp: Slot [28] registered [ 0.303123] acpiphp: Slot [29] registered [ 0.305095] acpiphp: Slot [30] registered [ 0.307152] acpiphp: Slot [31] registered [ 0.312114] PCI host bridge to bus 0000:00 [ 0.316033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337080] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340174] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350881] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370262] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373970] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473671] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.520018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.554015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.566881] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.573014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.676019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.741814] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.753000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.770014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.804058] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.825106] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.830013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.836013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.863014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.878236] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.883013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.890012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.915014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.929619] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.935464] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.943174] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.947524] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.950352] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.953658] iommu: Default domain type: Passthrough [ 0.955347] SCSI subsystem initialized [ 0.957328] ACPI: bus type USB registered [ 0.959096] usbcore: registered new interface driver usbfs [ 0.962088] usbcore: registered new interface driver hub [ 0.965102] usbcore: registered new device driver usb [ 0.967279] pps_core: LinuxPPS API ver. 1 registered [ 0.969011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.973057] PTP clock support registered [ 0.976452] EDAC MC: Ver: 3.0.0 [ 0.977122] PCI: Using ACPI for IRQ routing [ 0.980131] NetLabel: Initializing [ 0.982014] NetLabel: domain hash size = 128 [ 0.984020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.987113] NetLabel: unlabeled traffic allowed by default [ 0.989470] vgaarb: loaded [ 0.991553] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.994017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.001359] clocksource: Switched to clocksource kvm-clock [ 1.155298] VFS: Disk quotas dquot_6.6.0 [ 1.157171] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.160202] *** VALIDATE ramfs *** [ 1.161792] *** VALIDATE hugetlbfs *** [ 1.163604] pnp: PnP ACPI init [ 1.166752] pnp: PnP ACPI: found 6 devices [ 1.183219] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.186743] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.192701] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.199251] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.203531] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.208380] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.214294] NET: Registered protocol family 2 [ 1.222826] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.230865] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.235840] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.244306] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.251559] TCP: Hash tables configured (established 65536 bind 65536) [ 1.254993] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.259442] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.262191] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.267341] NET: Registered protocol family 1 [ 1.270495] RPC: Registered named UNIX socket transport module. [ 1.272664] RPC: Registered udp transport module. [ 1.274373] RPC: Registered tcp transport module. [ 1.275946] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.278174] NET: Registered protocol family 44 [ 1.279609] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.281528] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.283538] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.285717] PCI: CLS 0 bytes, default 64 [ 1.287321] Unpacking initramfs... [ 3.300402] debug: unmapping init [mem 0xffff9658bcc54000-0xffff9658bffbffff] [ 3.304435] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.306517] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.309725] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.191397] Initialise system trusted keyrings [ 4.192764] Key type blacklist registered [ 4.195433] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.202911] zbud: loaded [ 4.205187] *** VALIDATE nfs *** [ 4.207767] *** VALIDATE nfs4 *** [ 4.209870] pstore: using deflate compression [ 4.213213] Platform Keyring initialized [ 4.327637] NET: Registered protocol family 38 [ 4.329377] Key type asymmetric registered [ 4.331116] Asymmetric key parser 'x509' registered [ 4.333135] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.336955] io scheduler mq-deadline registered [ 4.345989] io scheduler kyber registered [ 4.347751] io scheduler bfq registered [ 4.356797] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.360983] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.364610] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.366981] ACPI: Power Button [PWRF] [ 4.371595] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.377985] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.391455] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.397229] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.410137] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.438918] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.467913] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.471766] Non-volatile memory driver v1.3 [ 4.473688] Linux agpgart interface v0.103 [ 4.506814] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 4.510322] vda: detected capacity change from 0 to 77004800 [ 4.532717] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.536778] vdb: detected capacity change from 0 to 1073741824 [ 4.574258] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.577776] vdc: detected capacity change from 0 to 2621440000 [ 4.599547] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.602896] vdd: detected capacity change from 0 to 2621440000 [ 4.617027] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.619171] vde: detected capacity change from 0 to 4294967296 [ 4.634448] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.636678] vdf: detected capacity change from 0 to 4294967296 [ 4.641396] libphy: Fixed MDIO Bus: probed [ 4.646992] usbcore: registered new interface driver usbserial_generic [ 4.648704] usbserial: USB Serial support registered for generic [ 4.651637] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.657656] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.660458] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.662973] mousedev: PS/2 mouse device common for all mice [ 4.670488] rtc_cmos 00:05: RTC can wake from S4 [ 4.676514] rtc_cmos 00:05: registered as rtc0 [ 4.680220] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.685788] intel_pstate: CPU model not supported [ 4.686812] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.695633] hid: raw HID events driver (C) Jiri Kosina [ 4.697461] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.699057] usbcore: registered new interface driver usbhid [ 4.703900] usbhid: USB HID core driver [ 4.704124] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.705843] drop_monitor: Initializing network drop monitor service [ 4.712368] Initializing XFRM netlink socket [ 4.714168] NET: Registered protocol family 10 [ 4.718873] Segment Routing with IPv6 [ 4.721166] NET: Registered protocol family 17 [ 4.724187] mpls_gso: MPLS GSO support [ 4.729801] RAS: Correctable Errors collector initialized. [ 4.732190] AVX version of gcm_enc/dec engaged. [ 4.733722] AES CTR mode by8 optimization enabled [ 4.818168] sched_clock: Marking stable (4818110690, 0)->(5975808901, -1157698211) [ 4.823648] registered taskstats version 1 [ 4.829342] Loading compiled-in X.509 certificates [ 4.834205] zswap: loaded using pool lzo/zbud [ 4.870426] Key type big_key registered [ 4.887624] Key type encrypted registered [ 4.890489] ima: No TPM chip found, activating TPM-bypass! [ 4.893764] ima: Allocated hash algorithm: sha1 [ 4.895800] ima: No architecture policies found [ 4.898399] evm: Initialising EVM extended attributes: [ 4.900977] evm: security.selinux [ 4.902577] evm: security.ima [ 4.904185] evm: security.capability [ 4.905895] evm: HMAC attrs: 0x1 [ 4.908754] rtc_cmos 00:05: setting system clock to 2026-09-22 18:01:36 UTC (1790100096) [ 4.916775] debug: unmapping init [mem 0xffffffff87c03000-0xffffffff87dfffff] [ 4.920985] debug: unmapping init [mem 0xffffffff86982000-0xffffffff86c58fff] [ 4.930334] Write protecting the kernel read-only data: 28672k [ 4.938826] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 4.942393] debug: unmapping init [mem 0xffffffff85914000-0xffffffff859fffff] [ 4.973228] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.979866] systemd[1]: Detected virtualization kvm. [ 4.981305] systemd[1]: Detected architecture x86-64. [ 4.983316] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.052358] systemd[1]: No hostname configured. [ 5.063737] systemd[1]: Set hostname to . [ 5.073298] random: systemd: uninitialized urandom read (16 bytes read) [ 5.082371] systemd[1]: Initializing machine ID from random generator. [ 5.316401] random: systemd: uninitialized urandom read (16 bytes read) [ 5.318376] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.322073] random: systemd: uninitialized urandom read (16 bytes read) [ 5.324024] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.329977] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.783947] device-mapper: uevent: version 1.0.3 [ 6.790669] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization.[ 8.682708] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.872559] virtio_net virtio0 ens2: renamed from eth0 [ 9.186027] scsi host0: ata_piix [ 9.412864] scsi host1: ata_piix [ 9.441626] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.448355] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.417047] random: crng init done [ 14.419677] random: 7 urandom warning(s) missed due to ratelimiting [ 17.837753] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.115702] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.211850] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.490415] SELinux: Disabled at runtime. [ 24.673604] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.710484] systemd[1]: Detected virtualization kvm. [ 24.714583] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.798367] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.803607] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.815843] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.821172] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.825280] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.847256] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.863965] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 27.323505] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.362906] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.141909] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.226611] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.997167] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.158849] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 34.440500] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.118632] NFS: Registering the id_resolver key type [ 35.124678] Key type id_resolver registered [ 35.127553] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 51.068048] hrtimer: interrupt took 6040219 ns [ 95.059594] libcfs: loading out-of-tree module taints kernel. [ 95.108355] Key type ._llcrypt registered [ 95.110231] Key type .llcrypt registered [ 95.251984] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 110.148553] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 111.293317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.310433] alg: No test for adler32 (adler32-zlib) [ 112.682733] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 113.387579] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 115.103232] Key type lgssc registered [ 116.568941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.697799] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.685265] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 168.561295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.598291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.821391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.846064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.957476] Lustre: lustre-MDT0000: new disk, initializing [ 170.037267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.055207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.061212] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.079262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.204554] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.243960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 183.244107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.262368] Lustre: Skipped 1 previous similar message [ 183.354940] Lustre: lustre-MDT0001: new disk, initializing [ 183.424183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.445667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.455718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.707229] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.435754] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.047229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.204299] Lustre: lustre-OST0000: new disk, initializing [ 197.208596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.212810] Lustre: 8441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.259882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.528110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.537195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.593178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.549820] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.818856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.910196] Lustre: lustre-OST0001: new disk, initializing [ 211.913675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.916739] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.969115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.091575] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.625402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.635469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.674173] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.140718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.724694] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.108773] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.291941] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 240.099980] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 241.725584] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 243.469634] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 244.847504] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 22 14:05:35 EDT 2026 [ 257.255879] Lustre: DEBUG MARKER: only running tests: 33-75 [ 258.357568] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 259.370346] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 273.890312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.891682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.898497] Lustre: Skipped 3 previous similar messages [ 273.903516] Lustre: Skipped 2 previous similar messages [ 279.010773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 279.013560] Lustre: Skipped 1 previous similar message [ 279.585361] Lustre: server umount lustre-MDT0000 complete [ 282.628124] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100374 with bad export cookie 14572562023582637505 [ 282.632694] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 282.638449] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 284.136315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.147073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 284.147180] Lustre: Skipped 1 previous similar message [ 288.997628] Lustre: server umount lustre-MDT0001 complete [ 292.177601] Lustre: server umount lustre-OST0000 complete [ 295.337718] Lustre: server umount lustre-OST0001 complete [ 301.085199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 309.125294] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 344.552255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.934625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.998647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 345.091729] Lustre: lustre-MDT0000: new disk, initializing [ 345.243042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.279740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.981383] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 368.267740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.457827] Lustre: 15110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 368.521907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 368.527846] Lustre: Skipped 1 previous similar message [ 368.623344] Lustre: lustre-MDT0001: new disk, initializing [ 368.747421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 368.775660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 368.784100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 374.952696] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 390.667807] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 398.928482] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 405.985157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.346971] Lustre: lustre-OST0000: new disk, initializing [ 406.354780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.366841] Lustre: 17083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.461943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.228674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.241555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.392424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 413.946992] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 426.277695] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 432.042333] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 432.432937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 438.267051] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 438.501242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 444.390399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.396682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.403594] LustreError: Skipped 1 previous similar message [ 444.405689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.425775] Lustre: Skipped 1 previous similar message [ 444.436194] Lustre: Skipped 4 previous similar messages [ 447.955143] Lustre: server umount lustre-OST0000 complete [ 454.624987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.635206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 454.669294] Lustre: Skipped 1 previous similar message [ 454.686819] Lustre: Skipped 1 previous similar message [ 459.178859] Lustre: server umount lustre-MDT0000 complete [ 459.748384] LustreError: 15117:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.774653] LustreError: 15117:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 463.442395] LustreError: 15104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100555 with bad export cookie 14572562023582639038 [ 463.447482] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.458798] LustreError: 15104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 463.986841] Lustre: server umount lustre-MDT0001 complete [ 468.099084] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:09:18 (1790100558) [ 471.131240] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 473.421265] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:09:23 (1790100563) [ 480.369948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.888527] LustreError: 19495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.028278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.854382] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 486.369436] LustreError: 19496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.386179] LustreError: 19496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 490.470978] LustreError: 19495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.650767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.611403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 507.530771] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.077171] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 519.849271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.376851] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 535.377744] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 546.784809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.794932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.809357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.369117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.800730] Lustre: server umount lustre-OST0000 complete [ 556.514680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.522062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.535156] Lustre: Skipped 1 previous similar message [ 562.721573] Lustre: server umount lustre-MDT0000 complete [ 565.730134] LustreError: 19496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.747368] LustreError: 19496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 566.142387] LustreError: 19476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100657 with bad export cookie 14572562023582639920 [ 566.144861] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.150833] LustreError: 19476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 566.438573] Lustre: server umount lustre-MDT0001 complete [ 576.792710] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 579.917921] Key type lgssc unregistered [ 580.309816] LNet: 23026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.321155] LNetError: 23026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.357175] LNet: Removed LNI 192.168.203.139@tcp [ 581.711194] Key type .llcrypt unregistered [ 581.722704] Key type ._llcrypt unregistered [ 604.782603] Key type ._llcrypt registered [ 604.784911] Key type .llcrypt registered [ 604.884376] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 606.590194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.628583] alg: No test for adler32 (adler32-zlib) [ 607.778531] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 608.086349] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 609.768099] Key type lgssc registered [ 611.320791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.121605] loop: module loaded [ 620.213282] loop0: detected capacity change from 0 to 204800000 [ 620.568483] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.592790] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.911554] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.076441] loop0: detected capacity change from 0 to 204800000 [ 628.355940] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.374930] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.403363] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.416097] loop0: detected capacity change from 0 to 204800000 [ 642.400484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.429248] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.827831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.884373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 644.008704] Lustre: lustre-MDT0000: new disk, initializing [ 644.135955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.181976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 648.484792] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 658.504701] loop1: detected capacity change from 0 to 204800000 [ 664.539753] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.872627] Lustre: lustre-OST07c6: new disk, initializing [ 664.879281] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 664.885695] Lustre: Skipped 1 previous similar message [ 664.889708] Lustre: 26236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 664.958807] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 672.718798] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 672.960460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 672.974765] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 673.025874] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 678.752524] Lustre: 25223:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 678.763883] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 678.782692] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 678.795654] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 678.805654] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 678.817456] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 691.641225] Lustre: 25224:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 691.650388] Lustre: 25224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.655485] Lustre: 25224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.665884] Lustre: 25224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 691.679806] Lustre: 25224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 691.693473] Lustre: 25224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.322399] Lustre: 25223:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 696.341866] Lustre: 25223:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 696.348615] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 696.357133] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.368401] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.379171] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.393651] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 696.406032] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.416818] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.425690] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.435869] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.448981] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.639811] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 699.649638] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.657721] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 699.666266] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.675677] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.686240] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.699599] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 699.705463] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.718311] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.730695] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.739989] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.753748] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.830919] Lustre: 25223:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 704.863718] Lustre: 25223:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.876137] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.881382] Lustre: 25223:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.893022] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.904648] Lustre: 25223:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.916584] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 704.939832] Lustre: 25223:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.948477] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 704.955735] Lustre: 25223:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.965306] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.976268] Lustre: 25223:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.082951] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 716.106656] Lustre: 25222:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 716.117921] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 716.131241] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.149731] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.164139] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.175132] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 716.192667] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.206837] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.228986] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.243819] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.249169] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 764.900567] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.918758] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 769.795731] Lustre: server umount lustre-OST07c6 complete [ 782.343396] Lustre: server umount lustre-MDT0000 complete [ 800.852922] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 803.429653] Key type lgssc unregistered [ 803.701130] LNet: 28787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.711814] LNetError: 28787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.725213] LNet: Removed LNI 192.168.203.139@tcp [ 804.665915] Key type .llcrypt unregistered [ 804.668161] Key type ._llcrypt unregistered [ 824.492300] Key type ._llcrypt registered [ 824.494880] Key type .llcrypt registered [ 824.606674] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 826.037284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.079744] alg: No test for adler32 (adler32-zlib) [ 827.211452] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 827.535344] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 829.208052] Key type lgssc registered [ 830.450686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.750312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.788161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.303095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.548345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 849.896688] LustreError: 30076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.931562] LustreError: 30076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 855.009971] LustreError: 30075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.236179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.633211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 860.743273] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 869.289927] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.513837] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 883.501021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.834342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.857338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 890.333950] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 898.654939] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 920.943398] Lustre: server umount lustre-OST0000 complete [ 927.206344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.213807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 927.231642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.262314] Lustre: Skipped 1 previous similar message [ 932.320905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.337227] Lustre: Skipped 1 previous similar message [ 936.612303] Lustre: server umount lustre-MDT0000 complete [ 937.441910] LustreError: 31053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.442109] Lustre: 29376:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101013/real 1790101013] req@000000003942dfeb x1877056861274368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790101029 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 937.476214] LustreError: 31053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 937.546903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.566022] Lustre: Skipped 1 previous similar message [ 940.844038] LustreError: 30057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101032 with bad export cookie 9463454459612768769 [ 940.850925] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.853962] LustreError: 30057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 941.289331] Lustre: server umount lustre-MDT0001 complete [ 953.849433] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 957.941143] Key type lgssc unregistered [ 958.262860] LNet: 33812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.278989] LNetError: 33812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.307189] LNet: Removed LNI 192.168.203.139@tcp [ 959.820170] Key type .llcrypt unregistered [ 959.824664] Key type ._llcrypt unregistered [ 981.723723] Key type ._llcrypt registered [ 981.726311] Key type .llcrypt registered [ 981.885361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 983.532781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.629202] alg: No test for adler32 (adler32-zlib) [ 984.673828] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 984.936697] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 986.671230] Key type lgssc registered [ 988.087304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 997.114729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 997.127673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.643795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.533343] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1009.123525] LustreError: 35097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.142153] LustreError: 35097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1012.577332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.199868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.951410] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1026.169800] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1032.510370] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1040.509511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.891088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.975631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1048.729708] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1057.195936] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1068.514151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.540291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1068.550531] Lustre: Skipped 1 previous similar message [ 1072.643632] Lustre: server umount lustre-OST0000 complete [ 1078.754636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.770782] Lustre: Skipped 1 previous similar message [ 1078.783126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1078.787568] Lustre: Skipped 1 previous similar message [ 1083.788129] Lustre: server umount lustre-MDT0000 complete [ 1083.875286] LustreError: 35826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.902032] LustreError: 35826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1089.000780] LustreError: 35091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.032067] LustreError: 35091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1089.671485] LustreError: 35079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101181 with bad export cookie 11670095565525253689 [ 1089.680724] LustreError: 35079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1089.681259] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.135880] Lustre: server umount lustre-MDT0001 complete [ 1104.215853] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1107.668188] Key type lgssc unregistered [ 1108.030755] LNet: 38632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.053186] LNetError: 38632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.091592] LNet: Removed LNI 192.168.203.139@tcp [ 1109.354388] Key type .llcrypt unregistered [ 1109.356282] Key type ._llcrypt unregistered [ 1134.949172] Key type ._llcrypt registered [ 1134.959064] Key type .llcrypt registered [ 1135.120203] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1136.190927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.334634] alg: No test for adler32 (adler32-zlib) [ 1137.529988] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 1137.852070] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1139.639207] Key type lgssc registered [ 1141.268757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1152.118657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1152.149359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1153.628635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.347721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1163.745440] LustreError: 40066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.779936] LustreError: 40066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1167.326478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.796604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1172.904688] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1181.287795] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.611653] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1195.913453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.280882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.349923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1203.649768] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1211.985411] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1219.042255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1219.054488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.078378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.941382] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 1223.141119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.146949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.171736] Lustre: Skipped 2 previous similar messages [ 1228.259739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.275706] Lustre: Skipped 3 previous similar messages [ 1233.379815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.381637] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1233.390762] Lustre: Skipped 3 previous similar messages [ 1233.604492] Lustre: server umount lustre-MDT0000 complete [ 1235.252227] LustreError: 40062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.870088] LustreError: 40047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101329 with bad export cookie 612395224246583695 [ 1237.877547] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.889476] LustreError: 40047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1238.498255] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.499475] LustreError: 40061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.518822] Lustre: Skipped 1 previous similar message [ 1242.536302] LustreError: 40061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.538879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1242.555792] LustreError: 40061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1242.572635] Lustre: Skipped 2 previous similar messages [ 1247.656842] LustreError: 40062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.678345] LustreError: 40062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1252.319433] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1252.574893] Lustre: server umount lustre-MDT0001 complete [ 1267.631445] Lustre: server umount lustre-OST0000 complete [ 1285.540710] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1288.653368] Key type lgssc unregistered [ 1288.889687] LNet: 43927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.897635] LNetError: 43927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.922150] LNet: Removed LNI 192.168.203.139@tcp [ 1290.060266] Key type .llcrypt unregistered [ 1290.062433] Key type ._llcrypt unregistered [ 1314.813495] Key type ._llcrypt registered [ 1314.815032] Key type .llcrypt registered [ 1314.936058] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1316.448472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.482718] alg: No test for adler32 (adler32-zlib) [ 1317.642780] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 1317.982805] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1319.735212] Key type lgssc registered [ 1321.218650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.477698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.492727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.090884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.198986] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1341.411271] LustreError: 45360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.436695] LustreError: 45360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1345.143733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.468826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.552682] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1359.482167] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.030352] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1374.034620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.551777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.554486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1380.862659] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1389.003232] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1397.069411] Lustre: lustre-OST0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 1397.221888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.230663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.288879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1400.297354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.301321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.325533] Lustre: Skipped 2 previous similar messages [ 1401.794827] Lustre: server umount lustre-OST0000 complete [ 1420.341254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.343503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.361975] Lustre: Skipped 1 previous similar message [ 1420.375395] Lustre: Skipped 1 previous similar message [ 1426.706294] Lustre: server umount lustre-MDT0000 complete [ 1426.919364] LustreError: 46090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.937172] LustreError: 45342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101522 with bad export cookie 17328723522790405896 [ 1430.946944] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.954252] LustreError: 45342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1445.343580] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.666620] Lustre: server umount lustre-MDT0001 complete [ 1456.551938] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1459.484668] Key type lgssc unregistered [ 1459.826804] LNet: 49071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.833822] LNetError: 49071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.857409] LNet: Removed LNI 192.168.203.139@tcp [ 1460.770466] Key type .llcrypt unregistered [ 1460.774662] Key type ._llcrypt unregistered [ 1483.587119] Key type ._llcrypt registered [ 1483.591063] Key type .llcrypt registered [ 1483.743408] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1484.705197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.720246] alg: No test for adler32 (adler32-zlib) [ 1485.748636] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 1485.973897] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1487.623268] Key type lgssc registered [ 1488.700139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1496.342655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1496.375176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.849738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.854445] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1508.321655] LustreError: 50502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.361483] LustreError: 50502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1511.131463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.482361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.688793] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1523.205463] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1528.154937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1534.988077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.336241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.662524] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1542.526429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1548.204766] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1553.552290] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1556.807465] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1569.007687] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-2214h27m39s [ 1570.656348] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1572.834554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1572.844272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.877907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.921928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.928819] Lustre: Skipped 4 previous similar messages [ 1575.749795] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 1579.049116] Lustre: server umount lustre-MDT0000 complete [ 1580.007085] LustreError: 52948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.018777] LustreError: 52948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1581.872765] LustreError: 50498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.654991] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1586.148864] LustreError: 50503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.163879] LustreError: 50503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1589.883987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.025795] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1590.287939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.345127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1591.340399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1595.819745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1599.926469] Lustre: DEBUG MARKER: Wait for df (36966) ... [ 1601.838738] Lustre: DEBUG MARKER: done [ 1606.635817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.648596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1606.676183] Lustre: Skipped 3 previous similar messages [ 1606.707377] Lustre: Skipped 2 previous similar messages [ 1611.759526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.769776] Lustre: Skipped 1 previous similar message [ 1612.260059] Lustre: server umount lustre-OST0000 complete [ 1616.868081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1616.883778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.913869] Lustre: Skipped 1 previous similar message [ 1623.031483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.036468] Lustre: Skipped 4 previous similar messages [ 1631.199555] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1631.553042] Lustre: server umount lustre-MDT0000 complete [ 1635.148547] LustreError: 50483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101726 with bad export cookie 17283822019605365165 [ 1635.172812] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.707943] Lustre: server umount lustre-MDT0001 complete [ 1648.267927] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1651.841318] Key type lgssc unregistered [ 1652.544686] LNet: 55432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.562340] LNetError: 55432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.601640] LNet: Removed LNI 192.168.203.139@tcp [ 1654.304265] Key type .llcrypt unregistered [ 1654.305899] Key type ._llcrypt unregistered [ 1698.224878] Key type ._llcrypt registered [ 1698.230158] Key type .llcrypt registered [ 1698.327767] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1699.789772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.816508] alg: No test for adler32 (adler32-zlib) [ 1700.914448] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 1701.162394] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1702.895913] Key type lgssc registered [ 1704.206818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1712.282534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1712.313224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.691105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1713.719415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1713.885319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.507258] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1727.877534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.024971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1728.066184] Lustre: 57653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1728.479382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1733.161300] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1741.988962] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.013683] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1755.237486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1755.473986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1755.681434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.068350] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1762.462678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1769.266988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1776.334228] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-2214h31m07s [ 1777.889652] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1784.864462] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1787.513318] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1787.567739] LustreError: 57661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1807.714803] Lustre: lustre-MDT0000: Client f479d9e6-6cda-474a-bb63-86907832b741 (at 192.168.203.39@tcp) reconnecting [ 1827.591171] LustreError: 57661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1829.951481] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1832.469275] Lustre: DEBUG MARKER: done [ 1839.075254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.088686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.107826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.597452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.607076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.624915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.839408] Lustre: server umount lustre-OST0000 complete [ 1846.240404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.249346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.268543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.402050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.414334] Lustre: Skipped 4 previous similar messages [ 1851.201231] Lustre: server umount lustre-MDT0000 complete [ 1854.856130] LustreError: 57647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101946 with bad export cookie 2259895421189372315 [ 1854.857087] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1854.867831] LustreError: 57647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1855.295700] Lustre: server umount lustre-MDT0001 complete [ 1866.105456] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1869.189903] Key type lgssc unregistered [ 1869.544820] LNet: 61980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.556224] LNetError: 61980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.605270] LNet: Removed LNI 192.168.203.139@tcp [ 1870.488737] Key type .llcrypt unregistered [ 1870.492153] Key type ._llcrypt unregistered [ 1915.618673] Key type ._llcrypt registered [ 1915.621795] Key type .llcrypt registered [ 1915.749977] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1916.731474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.823099] alg: No test for adler32 (adler32-zlib) [ 1917.824602] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 1918.047063] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1919.679205] Key type lgssc registered [ 1920.853714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1924.337753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1924.355874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.724238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1925.786463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.898505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.144522] Lustre: Failing over lustre-MDT0000 [ 1927.514937] Lustre: server umount lustre-MDT0000 complete [ 1934.824157] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:33:45 (1790102025) [ 1942.121898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.817102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.784345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1957.550520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.732587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1957.792494] Lustre: 64794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1958.068211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.276173] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1971.838937] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.918902] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.374082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.646855] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1985.966467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.173238] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1995.433320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2001.189872] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2006.214736] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2014.689409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.699631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.709274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.226391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.230171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.237812] Lustre: Skipped 1 previous similar message [ 2016.262461] Lustre: Skipped 1 previous similar message [ 2019.126207] Lustre: server umount lustre-MDT0000 complete [ 2023.009253] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102114 with bad export cookie 7681194976344392601 [ 2023.012077] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.022250] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2023.287574] Lustre: server umount lustre-MDT0001 complete [ 2026.557549] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2031.288530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2039.769748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.848720] Lustre: 63707:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102118/real 1790102118] req@00000000e3f30e4e x1877058004231168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.880882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.896957] Lustre: 63706:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102120/real 1790102120] req@00000000892575a7 x1877058004231296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.967213] Lustre: 63709:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102123/real 1790102123] req@00000000b12acb86 x1877058004231424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2048.038964] LustreError: 63705:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006ce6f0e5 x1877058004232320/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2048.510910] LustreError: 66644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.536624] LustreError: 66644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2048.656415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.766583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2052.647112] LustreError: 66645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.563373] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2053.671063] LustreError: 68567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.698498] LustreError: 68567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2054.624826] Lustre: 63706:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102129/real 1790102129] req@000000004ba106d6 x1877058004231808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2054.653033] Lustre: 63706:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2058.728641] LustreError: 68566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.746728] LustreError: 68566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2059.744147] Lustre: 63706:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102134/real 1790102134] req@000000007f0b15aa x1877058004231936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2062.457195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.157854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.951772] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2068.471789] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2071.614148] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.938639] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2082.143650] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2092.001267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.005828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.027208] Lustre: Skipped 2 previous similar messages [ 2092.047308] Lustre: Skipped 3 previous similar messages [ 2095.391338] Lustre: server umount lustre-MDT0000 complete [ 2097.132302] LustreError: 68566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.160798] LustreError: 68566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2099.207310] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102190 with bad export cookie 7681194976344394876 [ 2099.213647] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.222825] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2099.421799] Lustre: server umount lustre-MDT0001 complete [ 2106.229088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2116.723758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.599215] Lustre: 63709:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790102193/real 1790102193] req@000000007dff29c5 x1877058004259456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790102209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.635394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.832374] LustreError: 63705:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c05f3c8 x1877058004260480/t0(0) o250->MGC192.168.203.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.280462] LustreError: 66644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.365592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.422513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2130.513344] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2141.442953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.216531] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2147.973831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2156.381575] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2161.472929] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2167.689742] Lustre: server umount lustre-MDT0000 complete [ 2167.779798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.784643] LustreError: 71250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.802649] Lustre: Skipped 2 previous similar messages [ 2167.850277] LustreError: 71250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 2171.634907] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102263 with bad export cookie 7681194976344396255 [ 2171.641259] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.645924] LustreError: 64786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2171.961863] Lustre: server umount lustre-MDT0001 complete [ 2179.087802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2182.569562] Lustre: DEBUG MARKER: files compared the same [ 2194.962591] Lustre: server umount lustre-OST0000 complete [ 2212.816646] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2215.717493] Key type lgssc unregistered [ 2216.023587] LNet: 74522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.029211] LNetError: 74522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.044527] LNet: Removed LNI 192.168.203.139@tcp [ 2216.938869] Key type .llcrypt unregistered [ 2216.941894] Key type ._llcrypt unregistered [ 2248.260633] Key type ._llcrypt registered [ 2248.265946] Key type .llcrypt registered [ 2248.393668] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2249.226938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.254235] alg: No test for adler32 (adler32-zlib) [ 2250.457738] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 2250.820575] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2252.567258] Key type lgssc registered [ 2253.986653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2262.521829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2262.552212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.085532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.606866] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2274.273124] LustreError: 75957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.300692] LustreError: 75957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2277.813908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.306430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.494055] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2291.253525] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2298.211550] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2305.044825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2305.247523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.360493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2311.384960] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug +malloc all [ 2319.371866] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2333.159762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.160361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.175232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.214731] Lustre: Skipped 1 previous similar message [ 2338.305396] LustreError: 77779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.503587] Lustre: server umount lustre-OST0000 complete [ 2343.393565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.414562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.419407] Lustre: Skipped 2 previous similar messages [ 2348.514211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.531126] Lustre: Skipped 1 previous similar message [ 2348.849208] Lustre: server umount lustre-MDT0000 complete [ 2352.550827] LustreError: 75936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102444 with bad export cookie 5883950182132074734 [ 2352.552773] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.570642] LustreError: 75936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2352.848229] Lustre: server umount lustre-MDT0001 complete [ 2365.311713] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2368.575465] Key type lgssc unregistered [ 2368.894678] LNet: 79510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.918341] LNetError: 79510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.961524] LNet: Removed LNI 192.168.203.139@tcp [ 2370.036920] Key type .llcrypt unregistered [ 2370.039610] Key type ._llcrypt unregistered [ 2394.807672] Key type ._llcrypt registered [ 2394.813710] Key type .llcrypt registered [ 2394.957877] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2395.918529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.999066] alg: No test for adler32 (adler32-zlib) [ 2397.112508] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 2397.366990] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2399.015343] Key type lgssc registered [ 2400.257417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2407.934665] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.961949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.519486] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2428.639335] LustreError: 80917:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 2455.391223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.566157] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2472.067399] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.753575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.887572] Lustre: *** cfs_fail_loc=706, val=0*** [ 2482.483497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.600174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2487.796091] LustreError: 80942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.829167] LustreError: 80942:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2487.976932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2489.838631] LustreError: 82002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.876437] LustreError: 82002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2494.963147] LustreError: 82002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.989238] LustreError: 82002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2499.204029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.785044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2506.048377] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2514.389628] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2521.615955] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2525.152293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2525.165590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.195235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.669950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.703224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2530.151858] Lustre: server umount lustre-OST0000 complete [ 2535.406363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2535.418576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.448333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.656724] Lustre: server umount lustre-MDT0000 complete [ 2541.027083] LustreError: 82007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.052900] LustreError: 82007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2544.892719] LustreError: 80924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102636 with bad export cookie 9193705613253936460 [ 2544.901079] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.902776] LustreError: 80924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2545.386926] Lustre: server umount lustre-MDT0001 complete [ 2557.381571] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2561.266482] Key type lgssc unregistered [ 2561.627569] LNet: 84532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.637969] LNetError: 84532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.653283] LNet: Removed LNI 192.168.203.139@tcp [ 2562.745508] Key type .llcrypt unregistered [ 2562.750247] Key type ._llcrypt unregistered [ 2594.425966] Key type ._llcrypt registered [ 2594.427851] Key type .llcrypt registered [ 2594.532001] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2595.846555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2595.874839] alg: No test for adler32 (adler32-zlib) [ 2596.930506] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 2597.338974] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2599.168826] Key type lgssc registered [ 2600.782650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2609.398604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2609.434551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.212709] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2625.456544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.113351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.141622] LustreError: 86650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.167192] LustreError: 86650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2631.451669] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2636.258469] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.287877] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2641.382167] LustreError: 86654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.411589] LustreError: 86654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2642.279979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.796362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.036584] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.070504] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2651.050680] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2656.228147] LustreError: 86654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.261900] LustreError: 86654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2662.055231] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.064772] Lustre: MGS: Client b716bf1c-0f4c-41cc-a997-73fbd3786c5d (at 0@lo) reconnecting [ 2662.081420] LustreError: 86664:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2662.334082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.431169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2667.545897] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2682.851584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.865601] Lustre: Skipped 1 previous similar message [ 2682.868861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.874634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2682.892445] Lustre: Skipped 1 previous similar message [ 2687.984344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.001645] Lustre: Skipped 1 previous similar message [ 2689.248398] Lustre: server umount lustre-OST0000 complete [ 2698.213296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.223549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.229495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.235479] Lustre: Skipped 1 previous similar message [ 2698.252691] Lustre: Skipped 1 previous similar message [ 2699.763197] Lustre: server umount lustre-MDT0000 complete [ 2703.329395] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.350435] LustreError: 86653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2704.479715] Lustre: server umount lustre-MDT0001 complete [ 2708.445610] Lustre: server umount MGS complete [ 2720.163890] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:46:50 (1790102810) [ 2749.542362] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2760.392355] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2799.649111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.985964] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2814.044574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.228777] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2814.300597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2814.405576] Lustre: lustre-MDT0001: new disk, initializing [ 2814.512090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.548592] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2814.558146] Lustre: 93844:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2814.573796] LustreError: 93844:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2819.482124] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2832.071514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.274697] Lustre: lustre-OST0000: new disk, initializing [ 2832.282481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2832.288223] Lustre: 94609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2832.370556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.880958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2840.672636] Lustre: 94943:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e3f30e4e x1877058716215680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790102948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2851.825257] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2860.881454] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.902602] Lustre: MGS: Client d47249a7-8ac5-431b-a53a-463f3c269ad2 (at 0@lo) reconnecting [ 2860.923640] LustreError: 93830:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2860.974643] Lustre: 93111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2861.022953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2861.025658] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2861.169979] Lustre: lustre-MDT0000: new disk, initializing [ 2861.394654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.416306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2862.510936] Lustre: 95679:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ed10154e x1877058716222848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790102970 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2866.982991] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2867.624653] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2871.690195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2871.718076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2871.744450] Lustre: Skipped 1 previous similar message [ 2871.765262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2899.429444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.436608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.454621] Lustre: Skipped 1 previous similar message [ 2899.472551] Lustre: Skipped 1 previous similar message [ 2904.561527] LustreError: 94595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.576881] LustreError: 94595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2904.713214] Lustre: server umount lustre-OST0000 complete [ 2909.664322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2909.670108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.679249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.691607] Lustre: Skipped 1 previous similar message [ 2909.711477] Lustre: Skipped 1 previous similar message [ 2914.852777] Lustre: server umount lustre-MDT0000 complete [ 2933.215484] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2933.975633] Lustre: server umount lustre-MDT0001 complete [ 2939.007672] Lustre: server umount MGS complete [ 2953.920632] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:50:44 (1790103044) [ 2974.807839] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2978.722743] Key type lgssc unregistered [ 2979.143718] LNet: 98181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.152797] LNetError: 98181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.170559] LNet: Removed LNI 192.168.203.139@tcp [ 2980.427985] Key type .llcrypt unregistered [ 2980.432755] Key type ._llcrypt unregistered [ 3001.263324] Key type ._llcrypt registered [ 3001.266821] Key type .llcrypt registered [ 3001.386678] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3002.527262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3002.544822] alg: No test for adler32 (adler32-zlib) [ 3003.646525] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3003.868088] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3005.596972] Key type lgssc registered [ 3007.041377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3014.853205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3014.881417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.323300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.124493] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3025.901755] LustreError: 99420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.923661] LustreError: 99420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3030.611541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.018164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3031.178945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3035.711255] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3045.245119] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3053.384542] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3063.310361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3063.718467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.541176] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3080.694310] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3086.335772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3096.553617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.561681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.562350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3096.562887] Lustre: Skipped 1 previous similar message [ 3096.597709] Lustre: Skipped 1 previous similar message [ 3099.907934] Lustre: server umount lustre-OST0000 complete [ 3106.792740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.805721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.806389] Lustre: Skipped 1 previous similar message [ 3106.821212] Lustre: Skipped 1 previous similar message [ 3110.904705] Lustre: server umount lustre-MDT0000 complete [ 3111.908934] LustreError: 101262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.929225] LustreError: 101262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3114.484044] LustreError: 99402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103206 with bad export cookie 8193061274864517925 [ 3114.487163] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.501057] LustreError: 99402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3115.174503] Lustre: server umount lustre-MDT0001 complete [ 3126.771324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3129.696879] Key type lgssc unregistered [ 3129.995834] LNet: 102978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.005841] LNetError: 102978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.024729] LNet: Removed LNI 192.168.203.139@tcp [ 3130.824917] Key type .llcrypt unregistered [ 3130.827436] Key type ._llcrypt unregistered [ 3146.864948] Key type ._llcrypt registered [ 3146.871074] Key type .llcrypt registered [ 3146.979714] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3147.961209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3148.072566] alg: No test for adler32 (adler32-zlib) [ 3149.270739] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3149.561302] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3151.303303] Key type lgssc registered [ 3152.560566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.656378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3158.671105] LustreError: 104076:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3158.677801] LustreError: 104079:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3158.694996] LustreError: 104076:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3158.709771] LustreError: 104076:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3158.723750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.725099] LustreError: 104076:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3160.214585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.657605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.148641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3173.955248] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3179.551985] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing lsmod [ 3182.597644] LustreError: 105285:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3182.620516] LustreError: 105286:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3182.629961] LustreError: 105285:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 3182.654616] LustreError: 105285:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3182.665090] LustreError: 105285:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3182.667719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.304653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.350076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3189.730610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3189.743869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.773535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.759699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.774293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.794338] Lustre: Skipped 1 previous similar message [ 3193.032575] Lustre: server umount lustre-MDT0000 complete [ 3193.902290] LustreError: 104110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.931148] LustreError: 104110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3196.874734] LustreError: 104094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103288 with bad export cookie 7259621588998013476 [ 3196.889781] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3198.111316] LustreError: 104547:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000653a37a1 x1877059296056832/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3198.315766] Lustre: server umount lustre-MDT0001 complete [ 3211.647544] Lustre: server umount lustre-OST0000 complete [ 3221.663627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.418214] LustreError: 106350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.563201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.620936] LustreError: 106351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.569864] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3232.736915] LustreError: 106350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.881494] LustreError: 106350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.907627] LustreError: 106350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3241.880036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.423814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.910065] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3257.365757] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3266.318540] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.996099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.572463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.667699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3289.172729] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.001792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3302.368462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.384360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.411400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.427726] Lustre: Skipped 2 previous similar messages [ 3304.417368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3304.431134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3304.436453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.478506] Lustre: Skipped 1 previous similar message [ 3306.870716] Lustre: server umount lustre-OST0000 complete [ 3314.145634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.157906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.170700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3317.035917] Lustre: server umount lustre-MDT0000 complete [ 3319.780094] LustreError: 106351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.797797] LustreError: 106351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3320.553934] LustreError: 106330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103412 with bad export cookie 7259621588998014421 [ 3320.560350] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.567662] LustreError: 106330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3320.949660] Lustre: server umount lustre-MDT0001 complete [ 3331.733926] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3334.127636] Key type lgssc unregistered [ 3334.460183] LNet: 109883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.479513] LNetError: 109883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.506520] LNet: Removed LNI 192.168.203.139@tcp [ 3335.178326] Key type .llcrypt unregistered [ 3335.180291] Key type ._llcrypt unregistered [ 3372.901132] Key type ._llcrypt registered [ 3372.906435] Key type .llcrypt registered [ 3373.110233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3374.568895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3374.584901] alg: No test for adler32 (adler32-zlib) [ 3375.827673] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3376.184633] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3378.007205] Key type lgssc registered [ 3379.729070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3390.684436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3390.721733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.160840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.105660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3401.698300] LustreError: 111317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.715620] LustreError: 111317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3406.819413] LustreError: 111316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.973290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.610661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.304636] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3420.892353] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3428.093259] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3435.851194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.285944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.310215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3443.512527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3451.438686] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3457.267678] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3460.719972] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3468.261251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.261688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.273526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3468.291261] Lustre: Skipped 1 previous similar message [ 3468.311932] Lustre: Skipped 1 previous similar message [ 3471.238392] Lustre: server umount lustre-OST0000 complete [ 3478.498334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.511685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.517652] Lustre: Skipped 1 previous similar message [ 3481.559218] Lustre: server umount lustre-MDT0000 complete [ 3483.617176] LustreError: 113912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.645149] LustreError: 113912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3485.183325] LustreError: 111297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103576 with bad export cookie 1006042540928349137 [ 3485.197706] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3485.206525] LustreError: 111297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3485.813228] Lustre: server umount lustre-MDT0001 complete [ 3496.077370] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3498.932122] Key type lgssc unregistered [ 3499.291985] LNet: 114956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3499.311319] LNetError: 114956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3499.334703] LNet: Removed LNI 192.168.203.139@tcp [ 3500.227396] Key type .llcrypt unregistered [ 3500.229845] Key type ._llcrypt unregistered [ 3517.430923] Key type ._llcrypt registered [ 3517.435328] Key type .llcrypt registered [ 3517.528806] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3518.276147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.308154] alg: No test for adler32 (adler32-zlib) [ 3519.347595] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3519.553861] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3521.215303] Key type lgssc registered [ 3522.447737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3529.172620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3529.201512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.552477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.336206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3540.962380] LustreError: 116194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.986705] LustreError: 116194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3542.167143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.578509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3547.483785] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3554.329439] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3559.866534] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.078132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.406455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.002718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3575.685240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3580.093576] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3588.067856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.072658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.081423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.099896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3589.114072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.132364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.150062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.168427] Lustre: server umount lustre-OST0000 complete [ 3598.816962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.821158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.840873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.846828] Lustre: Skipped 1 previous similar message [ 3601.386039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.883103] Lustre: server umount lustre-MDT0000 complete [ 3606.157380] LustreError: 116173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103697 with bad export cookie 15311868325877297885 [ 3606.166669] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.169319] LustreError: 116173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3606.445960] Lustre: server umount lustre-MDT0001 complete [ 3614.572882] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3617.432320] Key type lgssc unregistered [ 3617.778695] LNet: 119737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3617.788903] LNetError: 119737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3617.817233] LNet: Removed LNI 192.168.203.139@tcp [ 3618.829955] Key type .llcrypt unregistered [ 3618.835229] Key type ._llcrypt unregistered [ 3639.784740] Key type ._llcrypt registered [ 3639.786993] Key type .llcrypt registered [ 3639.899553] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3640.867511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3640.884769] alg: No test for adler32 (adler32-zlib) [ 3641.927210] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3642.159694] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3643.815589] Key type lgssc registered [ 3644.832422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3651.508438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3651.528794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.977782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.511609] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3663.336498] LustreError: 121147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.352369] LustreError: 121147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3663.399780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.762124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3667.457725] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3673.183368] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3678.603064] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3685.861529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.188949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.860685] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3697.457860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3698.628428] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.772512] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3708.165816] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3708.171145] Lustre: Skipped 2 previous similar messages [ 3711.800327] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3711.809306] Lustre: Skipped 2 previous similar messages [ 3713.105613] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3713.134770] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3718.153722] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3718.168907] Lustre: Skipped 2 previous similar messages [ 3724.837969] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.39@tcp 192.168.0.[2,10]@tcp [ 3728.493455] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.39/24@tcp in log lustre-MDT0000 [ 3728.498497] Lustre: Skipped 2 previous similar messages [ 3733.550171] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.39/24@tcp [ 3733.555495] Lustre: Skipped 1 previous similar message [ 3740.641372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.650561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.665830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.166978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3741.174656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.177355] LustreError: Skipped 1 previous similar message [ 3741.209421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3743.724810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3743.733388] Lustre: Skipped 1 previous similar message [ 3745.575479] Lustre: server umount lustre-OST0000 complete [ 3748.833090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.836058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.846956] Lustre: Skipped 1 previous similar message [ 3753.956034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.967237] Lustre: Skipped 2 previous similar messages [ 3754.682668] Lustre: server umount lustre-MDT0000 complete [ 3757.486290] LustreError: 121128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103849 with bad export cookie 3137089121219565698 [ 3757.488170] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.492922] LustreError: 121128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3757.782309] Lustre: server umount lustre-MDT0001 complete [ 3766.005609] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3768.245029] Key type lgssc unregistered [ 3768.524658] LNet: 126077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.532513] LNetError: 126077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.550174] LNet: Removed LNI 192.168.203.139@tcp [ 3769.177494] Key type .llcrypt unregistered [ 3769.179706] Key type ._llcrypt unregistered [ 3789.312430] Key type ._llcrypt registered [ 3789.314654] Key type .llcrypt registered [ 3789.393478] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3790.053538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3790.105306] alg: No test for adler32 (adler32-zlib) [ 3791.090452] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3791.268477] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3792.918202] Key type lgssc registered [ 3793.829285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.989840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3800.020859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.474611] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3801.475797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.498261] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.39/24@tcp [ 3805.330760] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3811.814328] LustreError: 127644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.832602] LustreError: 127644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3812.503045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.813736] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3812.833442] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.39/24@tcp [ 3812.876421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3816.179744] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3822.280988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3827.112738] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3832.883032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.176413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.610249] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3843.171851] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3846.532384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3856.870155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.882427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.883400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.884575] Lustre: Skipped 1 previous similar message [ 3860.995278] Lustre: server umount lustre-OST0000 complete [ 3864.036577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3864.041263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.058483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3864.064692] Lustre: Skipped 1 previous similar message [ 3867.106210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.107512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3867.129633] Lustre: Skipped 1 previous similar message [ 3872.224555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3872.228065] Lustre: Skipped 1 previous similar message [ 3877.345294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.354439] Lustre: Skipped 1 previous similar message [ 3878.369244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3878.596083] Lustre: server umount lustre-MDT0000 complete [ 3882.470758] LustreError: 127640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.501292] LustreError: 127640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3882.527574] LustreError: 127624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103974 with bad export cookie 954924203298939566 [ 3882.535355] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.557319] LustreError: 127624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3883.062839] Lustre: server umount lustre-MDT0001 complete [ 3904.103507] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3909.090395] Key type lgssc unregistered [ 3909.791212] LNet: 131259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.808737] LNetError: 131259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.870643] LNet: Removed LNI 192.168.203.139@tcp [ 3911.169292] Key type .llcrypt unregistered [ 3911.171336] Key type ._llcrypt unregistered [ 3948.297805] Key type ._llcrypt registered [ 3948.302317] Key type .llcrypt registered [ 3948.468602] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3963.258639] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3964.243680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3964.326901] alg: No test for adler32 (adler32-zlib) [ 3965.421380] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 3965.715093] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3967.375966] Key type lgssc registered [ 3968.888364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4003.138256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4003.166232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.462214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4004.511713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4004.615447] Lustre: lustre-MDT0000: new disk, initializing [ 4004.749517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.779310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4009.224396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4019.407084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.509204] Lustre: 134683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4019.569544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4019.573374] Lustre: Skipped 1 previous similar message [ 4019.637814] Lustre: lustre-MDT0001: new disk, initializing [ 4019.715111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.747074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4019.760851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4023.903487] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4032.354721] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4038.139848] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4044.801693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4044.956186] Lustre: lustre-OST0000: new disk, initializing [ 4044.959579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4044.965259] Lustre: 136648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4045.067598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.590540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4051.606454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4051.793987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4052.429466] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4062.610625] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4067.769528] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4067.974989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4073.857451] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4074.200817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4077.550492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4077.550977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.640648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.646755] Lustre: server umount lustre-OST0000 complete [ 4087.778074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.818079] Lustre: Skipped 1 previous similar message [ 4087.824201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4087.830898] Lustre: Skipped 2 previous similar messages [ 4092.900463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4092.908430] Lustre: Skipped 1 previous similar message [ 4093.424372] Lustre: server umount lustre-MDT0000 complete [ 4097.496489] LustreError: 134673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104189 with bad export cookie 5225748872662208281 [ 4097.505056] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.515592] LustreError: 134673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4097.855242] Lustre: server umount lustre-MDT0001 complete [ 4107.469337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.131145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.006447] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4113.376199] LustreError: 138863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.387909] LustreError: 138863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4117.474950] LustreError: 138864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.574869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.030814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.594685] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4134.175803] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.552190] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4147.453595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4147.851956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.984593] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4161.731343] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4166.892674] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4167.141282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4171.873352] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4172.085302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4182.613293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.940492] Lustre: lustre-OST0001: new disk, initializing [ 4182.948877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4182.960282] Lustre: 142235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4183.111887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.178984] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4191.416868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4191.439710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4191.716402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4201.361790] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4201.691328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4208.321252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4208.711259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.835164] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92b0057b5000.ost_server_uuid 50 [ 4216.900949] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92b0057b5000.ost_server_uuid in IDLE state after 0 sec [ 4227.560665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.571595] LustreError: Skipped 1 previous similar message [ 4227.576826] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.581039] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4227.592863] Lustre: Skipped 2 previous similar messages [ 4232.677689] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4232.692419] Lustre: Skipped 2 previous similar messages [ 4232.984163] Lustre: server umount lustre-OST0001 complete [ 4237.801906] LustreError: 140686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.817242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.842066] LustreError: 140686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4237.879385] Lustre: Skipped 1 previous similar message [ 4242.912648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.916434] Lustre: Skipped 2 previous similar messages [ 4242.928700] LustreError: 140684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.946585] LustreError: 140684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4243.432327] Lustre: server umount lustre-OST0000 complete [ 4248.033247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.988640] Lustre: server umount lustre-MDT0000 complete [ 4258.288351] LustreError: 140696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.317219] LustreError: 140696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4258.506683] LustreError: 138846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104350 with bad export cookie 5225748872662209170 [ 4258.508749] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.531825] LustreError: 138846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4258.929515] Lustre: server umount lustre-MDT0001 complete [ 4279.842618] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4282.666406] Key type lgssc unregistered [ 4282.931611] LNet: 145240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4282.941731] LNetError: 145240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4282.952940] LNet: Removed LNI 192.168.203.139@tcp [ 4283.805321] Key type .llcrypt unregistered [ 4283.810542] Key type ._llcrypt unregistered [ 4324.972738] Key type ._llcrypt registered [ 4324.975792] Key type .llcrypt registered [ 4325.096682] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4340.206776] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4341.153683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4341.179255] alg: No test for adler32 (adler32-zlib) [ 4342.396585] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 4342.666566] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4344.399306] Key type lgssc registered [ 4345.901550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4379.279931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4379.304438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.607107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4380.642162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4380.729703] Lustre: lustre-MDT0000: new disk, initializing [ 4380.861671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.887753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4385.092591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4396.155925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.363983] Lustre: 149411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4396.491885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4396.497833] Lustre: Skipped 1 previous similar message [ 4396.610637] Lustre: lustre-MDT0001: new disk, initializing [ 4396.710847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.755514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4396.771315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4401.637616] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4411.241355] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4417.710074] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.459298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4425.885139] Lustre: lustre-OST0000: new disk, initializing [ 4425.910157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4425.922453] Lustre: 151382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4426.051049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.940879] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4433.530051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4433.548140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4433.715309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4442.609912] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4447.045558] Lustre: Failing over lustre-OST0000 [ 4447.365509] Lustre: server umount lustre-OST0000 complete [ 4464.607106] Lustre: 147626:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104540/real 1790104540] req@000000001dbe672f x1877060547020416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790104556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4464.613248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.625755] Lustre: 147626:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4464.648183] Lustre: Skipped 1 previous similar message [ 4466.598563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.978757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.009213] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4469.984108] Lustre: 147623:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104545/real 1790104545] req@000000004fd72493 x1877060547021696/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790104561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4470.016367] Lustre: 147623:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4474.381956] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4475.114994] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4475.233712] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4475.235760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4476.319150] Lustre: 147623:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104551/real 1790104551] req@000000002eb284ce x1877060547022720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790104567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4476.351346] Lustre: 147623:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4478.660823] Lustre: Failing over lustre-MDT0000 [ 4478.946668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4478.950063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.115335] Lustre: server umount lustre-MDT0000 complete [ 4480.483364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.517702] Lustre: Skipped 1 previous similar message [ 4485.428406] LustreError: 149418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.448289] LustreError: 149418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4490.545437] LustreError: 149416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.577039] LustreError: 149416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4495.659941] LustreError: 151289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4495.699104] LustreError: 151289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4496.736196] Lustre: 147626:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790104572/real 1790104572] req@00000000458b5b4a x1877060547033344/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790104588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4496.770047] Lustre: 147626:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4496.777811] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.645571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.655343] LustreError: 149418:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.678878] LustreError: 149418:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4506.797474] LustreError: 149416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.829619] LustreError: 149416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4507.465849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.528594] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4512.044744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4512.161239] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4512.751470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4512.757284] Lustre: Skipped 1 previous similar message [ 4512.780742] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4512.841812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4521.440770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4521.452417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.475543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.988830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.995816] Lustre: Skipped 1 previous similar message [ 4526.811784] Lustre: server umount lustre-OST0000 complete [ 4533.216830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.233962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4533.239612] Lustre: Skipped 1 previous similar message [ 4533.251327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.806464] Lustre: server umount lustre-MDT0000 complete [ 4538.339155] LustreError: 150575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.359437] LustreError: 150575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4540.447444] LustreError: 150216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104632 with bad export cookie 10557797397087151699 [ 4540.454139] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.458973] LustreError: 150216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4540.845496] Lustre: server umount lustre-MDT0001 complete [ 4550.832146] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4553.765297] Key type lgssc unregistered [ 4554.104679] LNet: 155043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4554.124648] LNetError: 155043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4554.152249] LNet: Removed LNI 192.168.203.139@tcp [ 4555.188540] Key type .llcrypt unregistered [ 4555.190513] Key type ._llcrypt unregistered [ 4594.048632] Key type ._llcrypt registered [ 4594.054786] Key type .llcrypt registered [ 4594.189286] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4595.871886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4595.889667] alg: No test for adler32 (adler32-zlib) [ 4596.912885] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 4597.213843] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4598.879289] Key type lgssc registered [ 4600.268715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4608.444770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4608.501752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.033040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.410483] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4620.258501] LustreError: 156582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.284736] LustreError: 156582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4624.833707] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.378672] LustreError: 156583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4629.389712] LustreError: 156611:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4629.400655] LustreError: 156611:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4629.412062] LustreError: 156611:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4630.019168] Lustre: server umount lustre-MDT0000 complete [ 4646.324082] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4656.580825] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4694.866234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.105597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4695.158702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4695.246963] Lustre: lustre-MDT0000: new disk, initializing [ 4695.344590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.367483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4699.875399] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4709.781374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.890195] Lustre: 159914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4709.961207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4709.965860] Lustre: Skipped 1 previous similar message [ 4710.045357] Lustre: lustre-MDT0001: new disk, initializing [ 4710.151119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.194704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4710.208548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4714.209980] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4723.215594] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.180242] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4734.516828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.753932] Lustre: lustre-OST0000: new disk, initializing [ 4734.758187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4734.764886] Lustre: 161881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4734.834165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.598406] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4741.720063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4741.734442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4741.861878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4749.135931] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4753.647298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4755.522805] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:20:46 (1790104846) [ 4991.968089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4991.970922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.999313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4992.993179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4992.994381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.011990] Lustre: Skipped 1 previous similar message [ 4993.047074] Lustre: Skipped 1 previous similar message [ 4997.622637] Lustre: server umount lustre-MDT0000 complete [ 4998.112273] LustreError: 159925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.128094] LustreError: 159925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5000.866942] LustreError: 159906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105092 with bad export cookie 15280058812332315323 [ 5000.874957] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5000.880312] LustreError: 159906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5003.236419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.236874] LustreError: 159919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.251262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5003.282947] Lustre: Skipped 1 previous similar message [ 5005.279846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5007.183578] Lustre: server umount lustre-MDT0001 complete [ 5017.339259] Lustre: server umount lustre-OST0000 complete [ 5025.076687] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5034.120948] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5069.953265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.291635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5070.375507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5070.649796] Lustre: lustre-MDT0000: new disk, initializing [ 5070.806768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.838055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5075.322430] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5084.661688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5084.791372] Lustre: 166537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5084.830419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5084.837133] Lustre: Skipped 1 previous similar message [ 5084.896769] Lustre: lustre-MDT0001: new disk, initializing [ 5084.996603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.042962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5085.052670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5088.635668] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5096.299387] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5101.560625] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5107.867656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5108.067275] Lustre: lustre-OST0000: new disk, initializing [ 5108.070635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5108.078074] Lustre: 168506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5108.152464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.082610] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5113.896467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5113.909572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5114.025975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5120.932756] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5124.724134] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5124.919467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5128.719173] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5128.932188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5134.822189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.823846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5134.844257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5134.857931] Lustre: Skipped 1 previous similar message [ 5145.055142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5145.057241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.070744] Lustre: Skipped 3 previous similar messages [ 5145.210945] Lustre: server umount lustre-OST0000 complete [ 5150.186153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.195464] Lustre: Skipped 2 previous similar messages [ 5155.213488] Lustre: server umount lustre-MDT0000 complete [ 5155.297219] LustreError: 166550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.312872] LustreError: 166550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5158.954770] LustreError: 166530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105250 with bad export cookie 15280058812332474153 [ 5158.968720] LustreError: 166530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5158.968808] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.429335] Lustre: server umount lustre-MDT0001 complete [ 5168.259837] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:27:39 (1790105259) [ 5185.797947] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5195.922391] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5228.002052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.160561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5228.193289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5228.269275] Lustre: lustre-MDT0000: new disk, initializing [ 5228.385209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.411120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5232.783632] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5243.644398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.751993] Lustre: 173262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5244.034396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5248.498148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5258.664445] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5264.497786] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5270.660693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.872918] Lustre: lustre-OST0000: new disk, initializing [ 5270.875739] Lustre: Skipped 1 previous similar message [ 5270.878840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5270.883265] Lustre: Skipped 2 previous similar messages [ 5270.887233] Lustre: 175236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5276.358396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5278.271688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5278.282842] Lustre: Skipped 1 previous similar message [ 5278.288706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5278.372810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5284.278907] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5294.051298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5294.055114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.061685] LustreError: Skipped 1 previous similar message [ 5294.062348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.062352] Lustre: Skipped 2 previous similar messages [ 5294.086516] Lustre: Skipped 1 previous similar message [ 5304.287570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.448642] Lustre: server umount lustre-OST0000 complete [ 5322.207106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5322.349175] Lustre: server umount lustre-MDT0000 complete [ 5324.769412] LustreError: 173273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.785793] LustreError: 173273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5325.356602] LustreError: 173253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105416 with bad export cookie 15280058812332475035 [ 5325.360217] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5325.373857] LustreError: 173253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5325.692975] Lustre: server umount lustre-MDT0001 complete [ 5332.324523] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:30:23 (1790105423) [ 5346.444833] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5357.486527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5390.983063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.166854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5391.189531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5391.257969] Lustre: lustre-MDT0000: new disk, initializing [ 5391.329703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.334512] Lustre: Skipped 2 previous similar messages [ 5391.350903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5394.762908] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5403.415994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.573826] Lustre: 179563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5403.885946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5408.567980] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5419.269534] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.405982] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5431.738579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5432.044619] Lustre: 181543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5437.908374] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5438.556809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5438.676155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5448.279257] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5459.429915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.431865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.448023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.448031] Lustre: Skipped 11 previous similar messages [ 5459.468578] Lustre: Skipped 3 previous similar messages [ 5459.518158] LustreError: Skipped 1 previous similar message [ 5463.737069] Lustre: server umount lustre-OST0000 complete [ 5474.792194] LustreError: 180705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.830622] LustreError: 180705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5474.833104] Lustre: server umount lustre-MDT0000 complete [ 5478.304889] LustreError: 181542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105569 with bad export cookie 15280058812332476120 [ 5478.315768] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.318988] LustreError: 181542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5478.696322] Lustre: server umount lustre-MDT0001 complete [ 5493.425310] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5496.547489] Key type lgssc unregistered [ 5496.965058] LNet: 183397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5496.971585] LNetError: 183397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5496.988626] LNet: Removed LNI 192.168.203.139@tcp [ 5497.978289] Key type .llcrypt unregistered [ 5497.980491] Key type ._llcrypt unregistered [ 5520.485484] Key type ._llcrypt registered [ 5520.488286] Key type .llcrypt registered [ 5520.588515] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5521.648555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.724639] alg: No test for adler32 (adler32-zlib) [ 5522.737286] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 5522.927968] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5524.616434] Key type lgssc registered [ 5525.763269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5532.568225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5532.584177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.000906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.296285] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5544.420236] LustreError: 184833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.438316] LustreError: 184833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5546.206168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.630229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.647942] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5558.049983] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5563.230794] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.789925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.211302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.111495] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5579.446097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5584.562183] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5605.348797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.358747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.379890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.801540] Lustre: server umount lustre-OST0000 complete [ 5615.592020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.597509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.599072] Lustre: Skipped 2 previous similar messages [ 5615.606471] Lustre: Skipped 1 previous similar message [ 5620.033096] Lustre: server umount lustre-MDT0000 complete [ 5620.707436] LustreError: 185560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.727330] LustreError: 185560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5623.849861] LustreError: 184814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105715 with bad export cookie 5767968144187322771 [ 5623.861448] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.129359] Lustre: server umount lustre-MDT0001 complete [ 5634.386989] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5636.856477] Key type lgssc unregistered [ 5637.149538] LNet: 188387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.157744] LNetError: 188387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.174582] LNet: Removed LNI 192.168.203.139@tcp [ 5637.961129] Key type .llcrypt unregistered [ 5637.963061] Key type ._llcrypt unregistered [ 5657.704725] Key type ._llcrypt registered [ 5657.706899] Key type .llcrypt registered [ 5657.818719] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5658.602694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5658.625036] alg: No test for adler32 (adler32-zlib) [ 5659.634398] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 5659.808240] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5661.447235] Key type lgssc registered [ 5662.433750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5669.411240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5669.453771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.026734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.378791] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5680.611789] LustreError: 189816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.636427] LustreError: 189816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5683.341639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.727243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5688.348693] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5695.026264] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5699.934726] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.409237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5707.722645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.512593] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5715.957600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5721.572243] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5728.595947] Lustre: lustre-OST0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 5729.251230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5729.264946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5729.285760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.299310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.302083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.325790] Lustre: Skipped 1 previous similar message [ 5733.548753] Lustre: server umount lustre-OST0000 complete [ 5741.117666] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92b018854000.ost_server_uuid 50 [ 5743.063122] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5756.896917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.909526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.916771] Lustre: Skipped 1 previous similar message [ 5762.781714] Lustre: server umount lustre-MDT0000 complete [ 5763.041645] LustreError: 189815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.064482] LustreError: 189815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5766.076781] LustreError: 189796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105857 with bad export cookie 12522802515872595729 [ 5766.085579] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5766.464451] Lustre: server umount lustre-MDT0001 complete [ 5774.652907] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:37:45 (1790105865) [ 5781.046424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.555209] LustreError: 193733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.629425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.980407] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5787.109778] LustreError: 193734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.236576] LustreError: 193733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.574512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.001880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5799.403337] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5807.709335] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5813.065585] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5819.962029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5820.315237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.942762] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5826.495952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5833.507604] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5840.362549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5840.561862] Lustre: lustre-OST0001: new disk, initializing [ 5840.565825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5840.573700] Lustre: 196573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5840.676438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.669234] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5847.671072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5847.701220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5847.801779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5860.224723] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5865.442242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.460928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.478338] Lustre: Skipped 1 previous similar message [ 5865.485215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.494706] Lustre: Skipped 2 previous similar messages [ 5867.497954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5869.879586] Lustre: server umount lustre-OST0000 complete [ 5871.405747] LustreError: 195555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.562497] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5876.756565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5881.025193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.266247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5889.004017] LustreError: 195556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.020096] LustreError: 195556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5899.237171] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.243310] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5899.263401] Lustre: Skipped 1 previous similar message [ 5899.267134] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5899.290477] Lustre: Skipped 5 previous similar messages [ 5901.859224] Lustre: server umount lustre-OST0001 complete [ 5907.939888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5907.957592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.978193] Lustre: Skipped 1 previous similar message [ 5911.703490] Lustre: server umount lustre-MDT0000 complete [ 5915.390082] LustreError: 193716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106006 with bad export cookie 12522802515872596800 [ 5915.404902] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5915.411177] LustreError: 193716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5915.886209] Lustre: server umount lustre-MDT0001 complete [ 5938.720863] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:40:29 (1790106029) [ 5945.867882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.149146] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5946.210764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5946.441125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.200141] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5962.567061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.643906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5962.678189] Lustre: 200015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5967.953593] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5974.589636] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5980.311049] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.535914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.786548] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5993.623713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5994.210736] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6001.465019] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.735524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.818338] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6015.565231] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6019.207170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6023.091834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6029.127854] Lustre: lustre-OST0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 6029.134681] Lustre: Skipped 3 previous similar messages [ 6029.282258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6029.297144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.314324] Lustre: Skipped 1 previous similar message [ 6031.991438] Lustre: server umount lustre-OST0000 complete [ 6034.224460] LustreError: 201865:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6034.258198] LustreError: 201865:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6049.760489] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6049.769039] LustreError: Skipped 1 previous similar message [ 6054.426516] Lustre: server umount lustre-OST0001 complete [ 6064.762595] Lustre: server umount lustre-MDT0000 complete [ 6068.770409] LustreError: 201896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106160 with bad export cookie 12522802515872598207 [ 6068.774533] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.785226] LustreError: 201896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6069.186579] Lustre: server umount lustre-MDT0001 complete [ 6091.058753] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:43:01 (1790106181) [ 6105.666727] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6114.943993] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6160.040536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.178524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6160.209619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6160.267571] Lustre: lustre-MDT0000: new disk, initializing [ 6160.391573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.400129] Lustre: Skipped 3 previous similar messages [ 6160.422042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6164.967755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6174.961324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.058311] Lustre: 208110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6175.115620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6175.118219] Lustre: Skipped 1 previous similar message [ 6175.174753] Lustre: lustre-MDT0001: new disk, initializing [ 6175.328810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6175.341832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6179.789765] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6189.182405] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6195.485834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6204.935412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.255305] Lustre: lustre-OST0000: new disk, initializing [ 6205.260823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6205.267449] Lustre: 210079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6214.752137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6214.788897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6214.988094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6216.789909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6227.125870] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6231.368726] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6231.552750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6235.693134] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6235.887332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6240.740203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.741569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6240.742653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6240.742658] Lustre: Skipped 7 previous similar messages [ 6240.748212] Lustre: Skipped 6 previous similar messages [ 6240.752275] LustreError: Skipped 2 previous similar messages [ 6243.863434] Lustre: server umount lustre-OST0000 complete [ 6253.652290] Lustre: server umount lustre-MDT0000 complete [ 6256.099535] LustreError: 208120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.124639] LustreError: 208120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6256.891935] LustreError: 208104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106348 with bad export cookie 12522802515872599985 [ 6256.894316] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.901191] LustreError: 208104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6257.164777] Lustre: server umount lustre-MDT0001 complete [ 6264.075389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.122877] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6276.603568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.802459] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6287.253699] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6292.645417] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6298.671680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.971396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.977255] Lustre: Skipped 4 previous similar messages [ 6303.944779] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6309.915951] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.670893] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6313.852716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6317.750602] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6317.890956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6320.610628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.617325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.621426] LustreError: Skipped 1 previous similar message [ 6320.627463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.638874] Lustre: Skipped 3 previous similar messages [ 6320.659631] Lustre: Skipped 4 previous similar messages [ 6325.484671] Lustre: server umount lustre-OST0000 complete [ 6331.052126] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6334.651686] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6362.751358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6367.759962] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6373.442437] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6382.148946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 6385.451125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.472344] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.488161] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6385.514459] Lustre: Skipped 1 previous similar message [ 6391.883704] Lustre: server umount lustre-OST0000 complete [ 6401.167733] Lustre: server umount lustre-MDT0000 complete [ 6403.965880] LustreError: 215409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106495 with bad export cookie 12522802515872600874 [ 6403.969883] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6403.982796] LustreError: 215409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6404.304418] Lustre: server umount lustre-MDT0001 complete [ 6410.950604] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:48:22 (1790106502) [ 6415.534731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.876232] LustreError: 218267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.885807] LustreError: 218267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6419.388449] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6426.410328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.034066] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6435.603794] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6440.334346] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6446.086116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.576787] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6457.047548] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.135959] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6461.341797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6464.526613] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6464.722355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6470.517947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6470.640417] Lustre: lustre-OST0001: new disk, initializing [ 6470.643869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6470.649579] Lustre: 221645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6475.530351] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6475.863643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6475.869867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6475.918114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6482.712905] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6486.232039] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6486.396519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6489.989442] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6490.202289] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6496.742813] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.745299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6496.749500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6496.749511] Lustre: Skipped 6 previous similar messages [ 6496.754359] Lustre: Skipped 5 previous similar messages [ 6496.761066] LustreError: Skipped 3 previous similar messages [ 6498.021160] Lustre: server umount lustre-OST0001 complete [ 6502.644539] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6502.790237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6506.240120] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6506.456776] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6534.056272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.922120] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6539.755784] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6539.771915] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6539.773629] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6539.784591] Lustre: Skipped 1 previous similar message [ 6544.528609] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6545.778632] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6553.167256] Lustre: server umount lustre-OST0001 complete [ 6573.792730] Lustre: server umount lustre-OST0000 complete [ 6583.162609] Lustre: server umount lustre-MDT0000 complete [ 6586.156490] LustreError: 218249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106677 with bad export cookie 12522802515872601840 [ 6586.159054] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.168477] LustreError: 218249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6592.486736] Lustre: server umount lustre-MDT0001 complete [ 6612.690440] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:51:43 (1790106703) [ 6618.796068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.056035] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6619.077269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6619.232970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.241199] Lustre: Skipped 6 previous similar messages [ 6623.712768] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6631.674716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.832053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6631.875428] Lustre: 226880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6636.442768] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6643.000502] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6647.635727] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6653.500170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.683963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6659.165831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6665.206988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6672.558280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6672.620260] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6677.283762] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6683.309844] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.500825] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6687.845313] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6688.021496] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6691.496741] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6691.688635] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6696.702471] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92b01132b800.ost_server_uuid 50 [ 6698.232936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92b01132b800.ost_server_uuid in FULL state after 0 sec [ 6699.310618] Lustre: Permanently deactivating lustre-OST0001 [ 6699.341179] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6702.360342] Lustre: Permanently reactivating lustre-OST0001 [ 6712.300147] Lustre: server umount lustre-OST0001 complete [ 6715.873489] LustreError: 230109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.893348] LustreError: 230109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 33 previous similar messages [ 6730.207239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6730.302987] Lustre: server umount lustre-OST0000 complete [ 6739.154337] LustreError: 231849:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000007e7fa763) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6741.260221] Lustre: server umount lustre-MDT0000 complete [ 6744.282655] LustreError: 226872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106835 with bad export cookie 12522802515872603121 [ 6744.284181] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.297394] LustreError: 226872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6744.546639] Lustre: server umount lustre-MDT0001 complete [ 6762.335603] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:54:13 (1790106853) [ 6767.393255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.581446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6767.608982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6767.617813] Lustre: Skipped 5 previous similar messages [ 6770.713350] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6776.670822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.756473] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6776.773258] Lustre: 233572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6780.067467] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6784.978427] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6788.882796] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6792.913232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6793.019370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6793.033340] Lustre: Permanently deactivating lustre-OST0000 [ 6793.040047] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6796.305782] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6800.472672] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6800.893383] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6804.923101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.980771] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6809.157949] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6809.743620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6814.565881] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6816.095945] Lustre: Permanently reactivating lustre-OST0000 [ 6816.117231] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6816.122387] Lustre: Skipped 2 previous similar messages [ 6819.368722] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6819.374209] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6819.386693] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6819.439287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6819.454216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6835.690231] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6835.691201] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.695848] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6835.695856] Lustre: Skipped 22 previous similar messages [ 6835.697489] LustreError: Skipped 8 previous similar messages [ 6835.712095] Lustre: Skipped 13 previous similar messages [ 6836.951320] Lustre: server umount lustre-OST0001 complete [ 6846.101105] Lustre: server umount lustre-OST0000 complete [ 6855.461232] Lustre: server umount lustre-MDT0000 complete [ 6858.299642] LustreError: 235439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106949 with bad export cookie 12522802515872604780 [ 6858.303053] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6858.313090] LustreError: 235439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6858.558591] Lustre: server umount lustre-MDT0001 complete [ 6866.084626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6867.991805] Key type lgssc unregistered [ 6868.236660] LNet: 238447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6868.244548] LNetError: 238447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6868.264434] LNet: Removed LNI 192.168.203.139@tcp [ 6868.910245] Key type .llcrypt unregistered [ 6868.912583] Key type ._llcrypt unregistered [ 6887.019627] Key type ._llcrypt registered [ 6887.021912] Key type .llcrypt registered [ 6887.102385] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6887.917213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6887.986346] alg: No test for adler32 (adler32-zlib) [ 6889.007392] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 6889.208798] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6890.855233] Key type lgssc registered [ 6891.837928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6899.625742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6899.649137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.982906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6904.074446] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6910.154149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.254379] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6910.290634] Lustre: 239897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6910.301099] Lustre: Permanently deactivating lustre-MDT0001 [ 6910.324308] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6910.434748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6910.627810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.904268] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6916.147628] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6920.096842] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6924.416884] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.609866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.903786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.893960] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6936.038176] LustreError: 241754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.056474] LustreError: 241754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6936.059445] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6941.154682] LustreError: 241754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.812423] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6946.275757] LustreError: 241755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.291038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6946.842814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.158821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6952.435218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6952.464405] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6952.480919] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6958.761073] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.354618] Lustre: Permanently reactivating lustre-MDT0001 [ 6962.390917] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6962.395900] Lustre: Skipped 2 previous similar messages [ 6964.294346] Lustre: lustre-MDT0001: Client 70bb11aa-ebf0-4fc5-a038-f9423e1d16ea (at 192.168.203.39@tcp) reconnecting [ 6967.870590] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.879182] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6967.883534] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6967.889816] LustreError: 243619:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6967.893839] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6967.896985] LustreError: 243619:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6967.897275] LustreError: 243619:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -108 [ 6967.897296] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6967.897296] [ 6967.898323] LustreError: 239901:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6968.988061] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6970.389701] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6974.758648] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6974.920796] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6978.304911] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6978.483084] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6980.851587] Lustre: Permanently deactivating lustre-MDT0001 [ 6980.865066] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6980.869465] Lustre: Skipped 2 previous similar messages [ 6988.954192] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7003.104325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.116720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.145665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7006.327311] Lustre: server umount lustre-MDT0000 complete [ 7008.745960] LustreError: 241023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.758684] LustreError: 241023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7009.501107] LustreError: 240648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107101 with bad export cookie 1912830733984381804 [ 7009.505580] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7009.515471] LustreError: 240648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7009.814147] Lustre: server umount lustre-MDT0001 complete [ 7015.362603] Lustre: server umount lustre-OST0000 complete [ 7025.266699] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:58:36 (1790107116) [ 7034.414349] Lustre: server umount lustre-OST0001 complete [ 7040.459101] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7048.369835] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7076.748390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.974474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7076.982410] Lustre: Skipped 2 previous similar messages [ 7077.002297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7077.055653] Lustre: lustre-MDT0000: new disk, initializing [ 7077.123565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.144180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7080.373136] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7088.284389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.371577] Lustre: 248211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7088.424906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7088.428597] Lustre: Skipped 1 previous similar message [ 7088.492081] Lustre: lustre-MDT0001: new disk, initializing [ 7088.583969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.624672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7088.640024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7091.848958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7099.270477] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7103.754618] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7108.888756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7109.092672] Lustre: lustre-OST0000: new disk, initializing [ 7109.095925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7109.101471] Lustre: 250180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7109.161371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.528735] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7116.359588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7116.374773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7116.435336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7120.817319] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7122.491182] LustreError: 248219:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7129.392902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.571989] Lustre: lustre-OST0001: new disk, initializing [ 7129.575235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7129.578601] Lustre: 251294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7132.568127] LustreError: 248219:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7132.593534] LustreError: 248221:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7134.594475] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7139.390121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7139.395205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7139.481962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7142.106305] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.660385] LustreError: 248221:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7143.360883] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7148.363193] Lustre: lustre-OST0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 7148.368285] Lustre: Skipped 3 previous similar messages [ 7150.049542] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7150.056263] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.067446] Lustre: Skipped 3 previous similar messages [ 7153.453378] Lustre: lustre-OST0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 7153.463192] Lustre: Skipped 2 previous similar messages [ 7158.582105] Lustre: lustre-OST0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 7158.591407] Lustre: Skipped 2 previous similar messages [ 7159.264838] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7159.334487] Lustre: server umount lustre-OST0001 complete [ 7160.291189] LustreError: 250164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.302044] LustreError: 250164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7170.536717] LustreError: 250164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.567934] LustreError: 250164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7175.654968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.672099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7175.679660] Lustre: Skipped 2 previous similar messages [ 7180.437991] Lustre: server umount lustre-OST0000 complete [ 7185.890624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7185.895117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.895943] LustreError: Skipped 1 previous similar message [ 7185.907805] Lustre: Skipped 1 previous similar message [ 7189.519693] Lustre: server umount lustre-MDT0000 complete [ 7191.008342] LustreError: 248225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.021807] LustreError: 248225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7192.469792] LustreError: 248206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107284 with bad export cookie 1912830733984384611 [ 7192.473239] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7192.479820] LustreError: 248206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7198.857645] Lustre: server umount lustre-MDT0001 complete [ 7207.899040] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7210.334428] Key type lgssc unregistered [ 7210.671203] LNet: 253337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7210.683769] LNetError: 253337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7210.708620] LNet: Removed LNI 192.168.203.139@tcp [ 7211.389556] Key type .llcrypt unregistered [ 7211.391250] Key type ._llcrypt unregistered [ 7245.027710] Key type ._llcrypt registered [ 7245.029651] Key type .llcrypt registered [ 7245.116438] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7245.919125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7245.948730] alg: No test for adler32 (adler32-zlib) [ 7246.976372] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 7247.234319] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7248.951271] Key type lgssc registered [ 7249.876463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7255.863217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7255.876742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.082915] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7257.107176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7257.241298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.905955] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7267.855147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.973699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7267.994396] Lustre: 255540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7268.217382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.774531] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7277.903629] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7282.159374] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7287.073067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7287.211475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7287.360304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.982209] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7297.110992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7298.144201] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7353.338429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7353.341407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.373857] LustreError: Skipped 1 previous similar message [ 7353.381477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.444561] Lustre: Skipped 1 previous similar message [ 7355.970663] Lustre: server umount lustre-OST0000 complete [ 7362.898735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7382.546189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7382.911877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.417159] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7394.173813] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7394.196238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7394.206449] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7394.218303] Lustre: Skipped 1 previous similar message [ 7397.639654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7440.352553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7440.353500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.361088] LustreError: Skipped 1 previous similar message [ 7440.366840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7440.366850] Lustre: Skipped 1 previous similar message [ 7440.407947] Lustre: Skipped 1 previous similar message [ 7443.729648] Lustre: server umount lustre-OST0000 complete [ 7450.592570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.624349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.634756] Lustre: Skipped 2 previous similar messages [ 7454.676842] Lustre: server umount lustre-MDT0000 complete [ 7458.694133] LustreError: 255532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107550 with bad export cookie 6561693005690853860 [ 7458.697764] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7458.710468] LustreError: 255532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7459.233973] Lustre: server umount lustre-MDT0001 complete [ 7474.344529] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7477.912298] Key type lgssc unregistered [ 7478.231872] LNet: 260962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.248702] LNetError: 260962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.264264] LNet: Removed LNI 192.168.203.139@tcp [ 7479.241633] Key type .llcrypt unregistered [ 7479.245506] Key type ._llcrypt unregistered [ 7504.965961] Key type ._llcrypt registered [ 7504.969282] Key type .llcrypt registered [ 7505.095930] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7505.957103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7506.021470] alg: No test for adler32 (adler32-zlib) [ 7507.401993] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 7507.712635] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7509.439223] Key type lgssc registered [ 7510.785501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7518.178929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7518.204887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.624234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.096547] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7524.836428] LustreError: 262396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.859560] LustreError: 262396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7529.963825] LustreError: 262397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7532.283638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.658136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.642752] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7545.892828] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7551.809986] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7559.086060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.457656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.594988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7565.214676] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7573.029470] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.681773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.690901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7603.703147] Lustre: Skipped 1 previous similar message [ 7608.008519] Lustre: server umount lustre-OST0000 complete [ 7613.920665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.947347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7613.959866] Lustre: Skipped 2 previous similar messages [ 7618.304676] Lustre: server umount lustre-MDT0000 complete [ 7619.048614] LustreError: 262397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7619.059352] LustreError: 262397:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7622.768087] LustreError: 262378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107714 with bad export cookie 16266690384600596591 [ 7622.772782] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7622.786417] LustreError: 262378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7623.470578] Lustre: server umount lustre-MDT0001 complete [ 7635.490584] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7638.521344] Key type lgssc unregistered [ 7638.888191] LNet: 266703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7638.907774] LNetError: 266703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7638.947568] LNet: Removed LNI 192.168.203.139@tcp [ 7639.848418] Key type .llcrypt unregistered [ 7639.861306] Key type ._llcrypt unregistered [ 7658.424184] Key type ._llcrypt registered [ 7658.431976] Key type .llcrypt registered [ 7658.551363] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7659.650788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7659.667518] alg: No test for adler32 (adler32-zlib) [ 7660.741987] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 7661.001363] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7662.696565] Key type lgssc registered [ 7664.069463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7671.254358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7671.272683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.829453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.530799] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7683.041761] LustreError: 267941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.056943] LustreError: 267941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7685.113741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.472766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.996587] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7698.497469] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7704.562184] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.038922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.359417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7717.496845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7718.803804] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7726.063819] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.386540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.402706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7740.410469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7740.424236] Lustre: Skipped 1 previous similar message [ 7745.508489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.516331] Lustre: Skipped 1 previous similar message [ 7746.551037] Lustre: server umount lustre-OST0000 complete [ 7752.161726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.172846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.182172] Lustre: Skipped 1 previous similar message [ 7752.189947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.751570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.754600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.776656] Lustre: Skipped 1 previous similar message [ 7757.125341] Lustre: server umount lustre-MDT0000 complete [ 7760.871114] LustreError: 267940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.895932] LustreError: 267940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7760.935829] LustreError: 267921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107852 with bad export cookie 5967603075553886085 [ 7760.947544] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.956678] LustreError: 267921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.247862] Lustre: server umount lustre-MDT0001 complete [ 7772.879694] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7776.648375] Key type lgssc unregistered [ 7777.056720] LNet: 271617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7777.066197] LNetError: 271617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7777.092026] LNet: Removed LNI 192.168.203.139@tcp [ 7778.250283] Key type .llcrypt unregistered [ 7778.253073] Key type ._llcrypt unregistered [ 7803.213186] Key type ._llcrypt registered [ 7803.214822] Key type .llcrypt registered [ 7803.320516] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7804.289398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7804.365748] alg: No test for adler32 (adler32-zlib) [ 7805.363560] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 7805.538073] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7807.191374] Key type lgssc registered [ 7808.300264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7815.261132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.292578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.796038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.440917] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7826.919165] LustreError: 273050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.953594] LustreError: 273050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7831.084345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.395623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7836.569293] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7845.456540] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.925773] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7859.298108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7859.806651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.884649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7866.162524] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7872.863954] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7902.177259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.183653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.190117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.201912] Lustre: Skipped 1 previous similar message [ 7904.407914] Lustre: server umount lustre-OST0000 complete [ 7908.321418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7908.331481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.344861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7908.351096] Lustre: Skipped 1 previous similar message [ 7912.424960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7912.429283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.309946] Lustre: server umount lustre-MDT0000 complete [ 7917.540649] LustreError: 273049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.565600] LustreError: 273049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7918.142324] LustreError: 273787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108009 with bad export cookie 9388736267884767868 [ 7918.148882] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7918.164741] LustreError: 273787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7918.518684] Lustre: server umount lustre-MDT0001 complete [ 7928.745629] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7931.135062] Key type lgssc unregistered [ 7931.351481] LNet: 277358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7931.363420] LNetError: 277358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7931.377075] LNet: Removed LNI 192.168.203.139@tcp [ 7932.044393] Key type .llcrypt unregistered [ 7932.046678] Key type ._llcrypt unregistered [ 7949.272370] Key type ._llcrypt registered [ 7949.275825] Key type .llcrypt registered [ 7949.385427] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7950.317749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7950.400344] alg: No test for adler32 (adler32-zlib) [ 7951.594830] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 7951.873755] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7953.535237] Key type lgssc registered [ 7954.703264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7961.985612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7962.000173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.499389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.282266] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7973.860195] LustreError: 278609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.889163] LustreError: 278609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7978.985883] LustreError: 278611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7979.227929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7979.633671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7984.482361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7991.410273] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7997.579964] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8004.089077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8004.473626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.565889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8010.811236] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8018.442799] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.152549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.162653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.172520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.194563] Lustre: Skipped 1 previous similar message [ 8034.280085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.287952] Lustre: Skipped 2 previous similar messages [ 8034.589691] Lustre: server umount lustre-OST0000 complete [ 8039.392511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.399026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.415068] Lustre: Skipped 1 previous similar message [ 8044.389852] Lustre: server umount lustre-MDT0000 complete [ 8044.516136] LustreError: 278596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.529581] LustreError: 278596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8048.988390] LustreError: 278576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108140 with bad export cookie 13002646117002045944 [ 8048.992379] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8049.008631] LustreError: 278576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8049.484256] Lustre: server umount lustre-MDT0001 complete [ 8060.552906] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8063.166634] Key type lgssc unregistered [ 8063.589129] LNet: 282307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8063.603311] LNetError: 282307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8063.627447] LNet: Removed LNI 192.168.203.139@tcp [ 8064.497879] Key type .llcrypt unregistered [ 8064.503729] Key type ._llcrypt unregistered [ 8077.306848] Key type ._llcrypt registered [ 8077.316230] Key type .llcrypt registered [ 8077.436527] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8096.154292] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8112.311285] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8113.095349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8113.110694] alg: No test for adler32 (adler32-zlib) [ 8114.181684] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 8114.395167] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8116.057956] Key type lgssc registered [ 8117.184542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.620378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8151.667086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.846780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8152.880320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8152.981650] Lustre: lustre-MDT0000: new disk, initializing [ 8153.072015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.112593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8156.884756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8167.205802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.334348] Lustre: 285875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8167.388322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8167.393399] Lustre: Skipped 1 previous similar message [ 8167.491378] Lustre: lustre-MDT0001: new disk, initializing [ 8167.582188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8167.612401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8167.628841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8172.461797] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8181.295547] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8186.039982] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8191.982564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.292681] Lustre: lustre-OST0000: new disk, initializing [ 8192.298783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8192.311970] Lustre: 287843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.401559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8198.211539] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8200.773246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8200.786497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8200.902262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8206.865328] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8210.914783] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8211.139520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8215.896342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8216.361367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8221.664289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8221.674706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.676282] LustreError: Skipped 1 previous similar message [ 8221.681954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8221.710247] Lustre: Skipped 1 previous similar message [ 8225.129626] Lustre: server umount lustre-OST0000 complete [ 8229.344676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8229.351697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8229.360706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.364613] Lustre: Skipped 1 previous similar message [ 8231.910737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8231.913381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.026913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.035912] Lustre: Skipped 2 previous similar messages [ 8242.143939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8242.159597] Lustre: Skipped 1 previous similar message [ 8243.167593] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8243.409126] Lustre: server umount lustre-MDT0000 complete [ 8248.202194] LustreError: 285867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108339 with bad export cookie 15288785808088044169 [ 8248.208746] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.215900] LustreError: 285867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8248.742898] Lustre: server umount lustre-MDT0001 complete [ 8258.146833] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:19:08 (1790108348) [ 8265.886932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.666095] LustreError: 290255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8266.694322] LustreError: 290255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8266.795483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.846101] LustreError: 290256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8272.011755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8276.960533] LustreError: 290255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8282.054711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.083096] LustreError: 290256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8282.444604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8287.029958] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8295.412356] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8301.479628] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.268177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.594059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.777097] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8324.631899] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8342.500483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.506682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.522178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.527061] Lustre: Skipped 1 previous similar message [ 8342.542610] Lustre: Skipped 1 previous similar message [ 8344.879362] Lustre: server umount lustre-OST0000 complete [ 8349.152279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8349.158973] LustreError: Skipped 1 previous similar message [ 8349.163410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.364967] Lustre: server umount lustre-MDT0000 complete [ 8357.861584] LustreError: 290255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.880841] LustreError: 290255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8359.044677] LustreError: 290236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108450 with bad export cookie 15288785808088045051 [ 8359.055359] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.057549] LustreError: 290236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8359.454949] Lustre: server umount lustre-MDT0001 complete [ 8371.202626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8374.636981] Key type lgssc unregistered [ 8374.967946] LNet: 293811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8374.980509] LNetError: 293811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8375.025202] LNet: Removed LNI 192.168.203.139@tcp [ 8376.165898] Key type .llcrypt unregistered [ 8376.167806] Key type ._llcrypt unregistered [ 8406.069982] Key type ._llcrypt registered [ 8406.074370] Key type .llcrypt registered [ 8406.186569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8420.182552] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8421.139660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8421.199719] alg: No test for adler32 (adler32-zlib) [ 8422.262269] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 8422.509586] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8424.175251] Key type lgssc registered [ 8425.190176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8468.637661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8468.657878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.903387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8469.941826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8470.024395] Lustre: lustre-MDT0000: new disk, initializing [ 8470.086230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.104339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8474.166900] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8484.372489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.520554] Lustre: 297653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8484.600275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8484.610122] Lustre: Skipped 1 previous similar message [ 8484.712394] Lustre: lustre-MDT0001: new disk, initializing [ 8484.792698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8484.836496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8484.844385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8489.746743] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8498.801386] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8504.067318] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8510.058050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8510.483069] Lustre: lustre-OST2710: new disk, initializing [ 8510.489470] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8510.503980] Lustre: 299620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8510.620586] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8516.369192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8518.190237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8518.199526] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8518.299134] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8525.134906] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8531.703944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8531.852334] Lustre: lustre-OST03e8: new disk, initializing [ 8531.859917] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8531.869994] Lustre: 300683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8531.982358] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8538.657758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8539.785305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8539.807221] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8539.898367] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8546.467485] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.814626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8555.051177] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8559.219426] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8559.483604] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8564.447184] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8564.702455] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8568.903348] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8569.072899] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8575.564446] Lustre: server umount lustre-MDT0000 complete [ 8575.968921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8576.001384] Lustre: Skipped 3 previous similar messages [ 8578.890300] LustreError: 297643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108670 with bad export cookie 366456954477568625 [ 8578.897805] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8578.899275] LustreError: 297643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8579.184411] Lustre: server umount lustre-MDT0001 complete [ 8592.660284] Lustre: server umount lustre-OST2710 complete [ 8596.959317] Lustre: 295437:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790108672/real 1790108672] req@0000000014cc45ed x1877064825242368/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790108688 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8597.007294] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.091490] Lustre: server umount lustre-OST03e8 complete [ 8617.730623] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8625.923832] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8660.892289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.352905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8661.418791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8661.576802] Lustre: lustre-MDT0000: new disk, initializing [ 8661.761335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.809214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8667.131921] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8678.882984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.045431] Lustre: 305932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8679.092793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8679.096818] Lustre: Skipped 1 previous similar message [ 8679.150448] Lustre: lustre-MDT0001: new disk, initializing [ 8679.247160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8679.276699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8679.297794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8684.341642] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8693.850769] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8698.842831] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8704.413979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8704.667075] Lustre: lustre-OST0000: new disk, initializing [ 8704.671444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8704.682844] Lustre: 307902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8704.769252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.960778] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8711.218657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8711.234430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8711.326353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8718.097078] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8721.804331] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8721.969877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8725.882827] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8726.063786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8732.131386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8732.131588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.138619] LustreError: Skipped 1 previous similar message [ 8732.140181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8732.172878] Lustre: Skipped 1 previous similar message [ 8733.889561] Lustre: server umount lustre-OST0000 complete [ 8737.253612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.257673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.268574] Lustre: Skipped 1 previous similar message [ 8737.274019] Lustre: Skipped 1 previous similar message [ 8742.368991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8742.379896] Lustre: Skipped 2 previous similar messages [ 8743.200917] Lustre: server umount lustre-MDT0000 complete [ 8745.970968] LustreError: 305925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108837 with bad export cookie 366456954477570305 [ 8745.977288] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.995579] LustreError: 305925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8746.270877] Lustre: server umount lustre-MDT0001 complete [ 8752.555948] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:27:23 (1790108843) [ 8753.594510] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8754.842458] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:27:25 (1790108845) [ 8757.524348] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8777.471988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.714655] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8777.740063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8777.888115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.258625] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8788.015162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.096396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8788.117593] Lustre: 311759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8791.753278] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8811.811800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8811.939612] Lustre: Denying initial registration attempt from nid 192.168.203.139@tcp, specified as failover [ 8811.950355] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8811.956092] LustreError: 313213:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8811.964355] LustreError: 313213:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8811.971138] LustreError: 313213:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8812.031946] Lustre: server umount lustre-OST0000 complete [ 8812.035363] LustreError: 313213:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8819.168747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8819.175524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.184812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.688137] Lustre: server umount lustre-MDT0000 complete [ 8825.126743] LustreError: 311749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108916 with bad export cookie 366456954477571187 [ 8825.134263] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8825.135028] LustreError: 311749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8825.401386] Lustre: server umount lustre-MDT0001 complete [ 8833.357850] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8835.434745] Key type lgssc unregistered [ 8835.669147] LNet: 314333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.673852] LNetError: 314333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8836.715516] LNet: Removed LNI 192.168.203.139@tcp [ 8837.368837] Key type .llcrypt unregistered [ 8837.371112] Key type ._llcrypt unregistered [ 8845.407393] Key type ._llcrypt registered [ 8845.411227] Key type .llcrypt registered [ 8845.492708] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8846.066891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.134305] alg: No test for adler32 (adler32-zlib) [ 8847.170890] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 8847.373723] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8849.007222] Key type lgssc registered [ 8849.902852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8885.863703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8891.270767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8891.281223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.463426] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8892.480563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8892.577650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.627167] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8901.214025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.300550] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8901.332507] Lustre: 317091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8901.503597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8904.232961] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8909.954567] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8913.849752] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8931.052151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8931.237601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8931.387954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.169916] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8939.411253] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8944.610802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.611201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.612953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8944.619520] LustreError: Skipped 1 previous similar message [ 8944.641775] Lustre: Skipped 1 previous similar message [ 8946.882387] Lustre: server umount lustre-OST0000 complete [ 8949.732037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.739427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.741507] Lustre: Skipped 1 previous similar message [ 8954.850390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.859714] Lustre: Skipped 1 previous similar message [ 8955.320969] Lustre: server umount lustre-MDT0000 complete [ 8957.454148] LustreError: 317083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109049 with bad export cookie 9751754250891979703 [ 8957.463979] LustreError: 319614:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 8957.464953] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.473445] LustreError: 319614:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 8957.692562] Lustre: server umount lustre-MDT0001 complete [ 8963.721216] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8965.445637] Key type lgssc unregistered [ 8965.625677] LNet: 320657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8965.633310] LNetError: 320657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8965.644286] LNet: Removed LNI 192.168.203.139@tcp [ 8966.167681] Key type .llcrypt unregistered [ 8966.169707] Key type ._llcrypt unregistered [ 8981.021396] Key type ._llcrypt registered [ 8981.022996] Key type .llcrypt registered [ 8981.083037] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8981.559387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.587056] alg: No test for adler32 (adler32-zlib) [ 8982.569738] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 8982.708267] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8984.343628] Key type lgssc registered [ 8985.103737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8990.020240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8990.035274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.284109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.591714] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8998.531218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.767688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9001.043619] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9005.169135] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.796990] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9014.957343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9015.206832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.693398] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9022.485439] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9034.040056] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 9034.723461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.724051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9034.725055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9034.729557] Lustre: Skipped 1 previous similar message [ 9038.786911] Lustre: server umount lustre-MDT0000 complete [ 9039.147569] LustreError: 322091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.157505] LustreError: 322091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9039.841959] LustreError: 322065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.852634] LustreError: 322065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9040.800834] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109132 with bad export cookie 8614512493275794015 [ 9040.802696] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.808029] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9044.270503] LustreError: 322062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.286987] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 9044.292319] Lustre: Skipped 2 previous similar messages [ 9044.961147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.968284] Lustre: Skipped 1 previous similar message [ 9049.385132] LustreError: 324754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.385194] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 9049.393308] LustreError: 324754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9049.403687] Lustre: Skipped 2 previous similar messages [ 9054.505833] LustreError: 322062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9054.506484] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 9054.517078] LustreError: 322062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9054.526790] Lustre: Skipped 1 previous similar message [ 9055.199177] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9055.355308] Lustre: server umount lustre-MDT0001 complete [ 9059.033725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9063.607685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.617322] LustreError: 323896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.663599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.710051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9067.839621] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9073.419128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.702459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9077.657816] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9080.886921] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9088.123842] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9092.337873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9100.645622] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9103.067707] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9109.478152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9109.488653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9109.496912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9111.521084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.295164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9123.565146] Lustre: server umount lustre-OST0000 complete [ 9129.952767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9129.967386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9129.997043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9129.999766] Lustre: Skipped 6 previous similar messages [ 9134.871139] Lustre: server umount lustre-MDT0000 complete [ 9135.016179] LustreError: 325658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9135.043876] LustreError: 325658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9139.389019] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109230 with bad export cookie 8614512493275801134 [ 9139.392988] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9139.403634] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9139.996974] Lustre: server umount lustre-MDT0001 complete [ 9152.518615] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9156.015709] Key type lgssc unregistered [ 9156.296789] LNet: 328330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9156.302977] LNetError: 328330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9156.318814] LNet: Removed LNI 192.168.203.139@tcp [ 9157.227666] Key type .llcrypt unregistered [ 9157.230817] Key type ._llcrypt unregistered [ 9185.884335] Key type ._llcrypt registered [ 9185.889270] Key type .llcrypt registered [ 9186.044676] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9187.369095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9187.413895] alg: No test for adler32 (adler32-zlib) [ 9188.454555] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 9188.803357] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9190.527370] Key type lgssc registered [ 9191.749977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9198.836349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9198.866341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9200.263256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.342539] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9209.827906] LustreError: 329762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.838417] LustreError: 329762:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9212.891735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.337551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9217.901284] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9225.386656] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9230.942528] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9234.913733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.914883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.936592] Lustre: Skipped 1 previous similar message [ 9234.958298] Lustre: Skipped 1 previous similar message [ 9238.052280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9238.055641] Lustre: Skipped 1 previous similar message [ 9240.337283] Lustre: server umount lustre-MDT0000 complete [ 9243.108491] LustreError: 329761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.126246] LustreError: 329761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9244.156110] LustreError: 329742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109335 with bad export cookie 17664633716981574473 [ 9244.175210] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.178554] LustreError: 329742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9244.673997] Lustre: server umount lustre-MDT0001 complete [ 9259.645416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.100789] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9260.152987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9260.389849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.392197] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9273.725713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.795927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9273.860941] Lustre: 332351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9274.192318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9279.033811] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9286.557575] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9291.933236] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9301.755482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.978593] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9301.989767] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9302.009582] LustreError: 334235:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9302.017159] LustreError: 334235:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9302.027191] LustreError: 334235:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9302.087827] Lustre: server umount lustre-OST0000 complete [ 9302.091294] LustreError: 334235:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9313.009166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9313.342759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9313.662094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9320.792488] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9321.078182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9329.191050] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9338.049970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.139188] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9338.227886] Lustre: lustre-OST0001: new disk, initializing [ 9338.231945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9338.238421] Lustre: 335773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9338.359341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9345.154041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9345.172035] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9345.303934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9345.659274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9354.233894] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9360.874247] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9360.875529] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.889136] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9360.893269] LustreError: Skipped 1 previous similar message [ 9360.908838] Lustre: Skipped 1 previous similar message [ 9363.327413] Lustre: server umount lustre-OST0001 complete [ 9365.989880] LustreError: 336091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.016966] LustreError: 336091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9367.537736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9367.559562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9367.582605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9367.591873] Lustre: Skipped 1 previous similar message [ 9371.107772] LustreError: 336091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.124550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9371.147196] LustreError: 336091:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9373.671772] Lustre: server umount lustre-OST0000 complete [ 9381.348927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.371958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.384113] Lustre: Skipped 3 previous similar messages [ 9384.466895] Lustre: server umount lustre-MDT0000 complete [ 9386.468360] LustreError: 332361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.506780] LustreError: 332361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9389.212916] LustreError: 333099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109480 with bad export cookie 17664633716981575019 [ 9389.223346] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9389.236181] LustreError: 333099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9389.541481] Lustre: server umount lustre-MDT0001 complete [ 9397.136471] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9399.211498] Key type lgssc unregistered [ 9399.442656] LNet: 337731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9399.448326] LNetError: 337731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9399.461460] LNet: Removed LNI 192.168.203.139@tcp [ 9399.980189] Key type .llcrypt unregistered [ 9399.982068] Key type ._llcrypt unregistered [ 9447.769612] Key type ._llcrypt registered [ 9447.771743] Key type .llcrypt registered [ 9447.843809] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9456.845805] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9457.528896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9457.554528] alg: No test for adler32 (adler32-zlib) [ 9458.515722] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 9458.702304] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9460.343289] Key type lgssc registered [ 9461.243835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9482.433781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9482.445221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.582588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9483.624430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9483.676453] Lustre: lustre-MDT0000: new disk, initializing [ 9483.727049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.738279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9486.133549] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9492.545038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.615769] Lustre: 342654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9492.658896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9492.662536] Lustre: Skipped 1 previous similar message [ 9492.716121] Lustre: lustre-MDT0001: new disk, initializing [ 9492.757525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9492.779886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9492.785104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9494.820875] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9500.267279] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9503.532902] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9507.448933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.602446] Lustre: lustre-OST0000: new disk, initializing [ 9507.604689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9507.609327] Lustre: 344621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9507.666066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.072995] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9514.003884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9514.010294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9514.100759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9517.310688] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9520.106306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9520.225398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9522.777207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9522.906345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9524.197098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9524.201018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.202437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.212086] Lustre: Skipped 1 previous similar message [ 9524.219464] Lustre: Skipped 1 previous similar message [ 9529.319795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9529.322714] Lustre: Skipped 1 previous similar message [ 9530.187495] Lustre: server umount lustre-OST0000 complete [ 9533.923549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9533.933149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.946412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9538.878850] Lustre: server umount lustre-MDT0000 complete [ 9540.963247] LustreError: 342644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109632 with bad export cookie 15922070656936711355 [ 9540.964302] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9540.973500] LustreError: 342644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9541.157987] Lustre: server umount lustre-MDT0001 complete [ 9545.868595] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:40:37 (1790109637) [ 9550.199950] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:40:41 (1790109641) [ 9555.311917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9555.561625] LustreError: 347372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.577983] LustreError: 347372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9555.618968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.211433] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9561.059894] LustreError: 347373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.651191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.961405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9566.694046] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9570.932880] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9573.914506] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9578.016694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.235676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.825187] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9586.274167] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9592.670203] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9593.731559] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9594.792921] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9595.917633] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9597.110837] Lustre: Failing over lustre-MDT0000 [ 9597.240610] Lustre: server umount lustre-MDT0000 complete [ 9598.778849] LustreError: 347369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.946186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9598.958483] Lustre: Skipped 3 previous similar messages [ 9603.889030] LustreError: 347368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.900515] LustreError: 347368:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9609.003565] LustreError: 347368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.018748] LustreError: 347368:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9613.223850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.323890] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.524490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9613.575823] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9614.137873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9616.746367] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9618.925685] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9618.950592] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9618.985246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9623.362506] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9624.350767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9626.626032] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9627.713814] Lustre: Failing over lustre-MDT0000 [ 9627.986421] Lustre: server umount lustre-MDT0000 complete [ 9629.157720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.159856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.161946] LustreError: 347369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.161957] LustreError: 347369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9629.167216] Lustre: Skipped 1 previous similar message [ 9644.178411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.259682] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.466702] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9644.847129] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9647.726637] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9649.649161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9649.657638] Lustre: Skipped 2 previous similar messages [ 9649.664462] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9649.692691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9654.110575] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9655.095876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9659.872362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9659.874209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.884463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.889885] Lustre: Skipped 2 previous similar messages [ 9659.908109] Lustre: Skipped 4 previous similar messages [ 9664.997754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9665.003453] Lustre: Skipped 2 previous similar messages [ 9665.496539] Lustre: server umount lustre-MDT0000 complete [ 9667.853751] LustreError: 347354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109759 with bad export cookie 15922070656936714050 [ 9667.858039] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9667.868891] LustreError: 347354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9668.033616] Lustre: server umount lustre-MDT0001 complete [ 9681.090818] Lustre: server umount lustre-OST0000 complete [ 9694.847829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.483961] LustreError: 353701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.514598] LustreError: 353701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 9695.669797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.690875] Lustre: Skipped 1 previous similar message [ 9700.072465] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9708.842990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.834591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9718.293564] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9721.658238] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9726.358028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9731.346383] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9736.535799] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9740.980673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9746.402292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.409230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9746.410347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9746.424701] Lustre: Skipped 1 previous similar message [ 9747.417804] Lustre: server umount lustre-OST0000 complete [ 9750.039835] Lustre: server umount lustre-MDT0000 complete [ 9750.497070] LustreError: 353701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.512351] LustreError: 353701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9752.489863] LustreError: 353683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109844 with bad export cookie 15922070656936714799 [ 9752.495242] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9752.499310] LustreError: 353683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9752.731725] Lustre: server umount lustre-MDT0001 complete [ 9759.930638] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 9761.880963] Key type lgssc unregistered [ 9762.088684] LNet: 357228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9762.095399] LNetError: 357228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9762.107088] LNet: Removed LNI 192.168.203.139@tcp [ 9762.716311] Key type .llcrypt unregistered [ 9762.718209] Key type ._llcrypt unregistered [ 9780.449176] Key type ._llcrypt registered [ 9780.451548] Key type .llcrypt registered [ 9780.543929] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9781.232496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9781.346284] alg: No test for adler32 (adler32-zlib) [ 9782.324979] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [ 9782.505114] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 9784.143591] Key type lgssc registered [ 9785.005286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9790.240545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9790.260872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.518107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9794.281169] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9799.383391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.602476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9802.196258] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9807.008098] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9812.422681] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9820.735297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.244328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9829.970601] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9837.055176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9840.633309] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9855.968244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9855.980686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9856.001623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.504107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.506376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.546279] Lustre: Skipped 1 previous similar message [ 9862.626603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.642594] Lustre: Skipped 4 previous similar messages [ 9867.746622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.756141] Lustre: Skipped 2 previous similar messages [ 9869.279170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9869.487189] Lustre: server umount lustre-MDT0000 complete [ 9872.868988] LustreError: 358743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.887052] LustreError: 358743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9873.418296] LustreError: 358723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109965 with bad export cookie 2817726294999587267 [ 9873.421982] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9873.435115] LustreError: 358723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9873.731354] Lustre: server umount lustre-MDT0001 complete [ 9890.946412] Lustre: server umount lustre-OST0000 complete [ 9911.886199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.277108] LustreError: 362478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.338826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9916.459124] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9917.427598] LustreError: 362479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.507172] LustreError: 362478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.158359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.434565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9928.465780] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9934.787236] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.609708] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9947.492409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9947.844879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.432782] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9957.046375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9961.934271] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9975.783651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9975.796452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.819799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9977.829047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.128150] Lustre: server umount lustre-MDT0000 complete [ 9982.951573] LustreError: 362479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.969207] LustreError: 362479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9983.763024] LustreError: 362459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110075 with bad export cookie 2817726294999588380 [ 9983.767071] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9983.774438] LustreError: 362459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9984.200688] Lustre: server umount lustre-MDT0001 complete [ 9998.670221] Lustre: server umount lustre-OST0000 complete [10014.166937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.718259] LustreError: 366390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.834651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10020.456982] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10031.315183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10037.075185] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10044.596799] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10049.902789] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10056.331761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10056.729051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10056.753736] Lustre: Skipped 1 previous similar message [10059.839738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10062.581919] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10070.450421] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10077.826334] Lustre: server umount lustre-OST0000 complete [10082.785227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10082.817348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.835759] Lustre: Skipped 1 previous similar message [10082.851357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10082.858551] Lustre: Skipped 3 previous similar messages [10091.584638] Lustre: server umount lustre-MDT0000 complete [10095.206263] LustreError: 368233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110186 with bad export cookie 2817726294999589402 [10095.217320] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10095.225578] LustreError: 368233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10104.958519] Lustre: server umount lustre-MDT0001 complete [10116.868436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10120.133575] Key type lgssc unregistered [10120.457728] LNet: 369923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10120.463231] LNetError: 369923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10120.478603] LNet: Removed LNI 192.168.203.139@tcp [10121.430438] Key type .llcrypt unregistered [10121.432586] Key type ._llcrypt unregistered [10149.245137] Key type ._llcrypt registered [10149.247531] Key type .llcrypt registered [10149.411211] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10150.600615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10150.628463] alg: No test for adler32 (adler32-zlib) [10151.688555] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [10151.989477] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10153.815197] Key type lgssc registered [10155.624167] Lustre: Echo OBD driver; http://www.lustre.org/ [10163.508312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10163.538232] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.548066] LustreError: 371384:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10163.577952] LustreError: 371384:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10163.587452] LustreError: 371384:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10163.597206] LustreError: 371384:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10163.604971] LustreError: 371384:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10171.505098] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10171.516731] LustreError: 371782:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10171.540860] LustreError: 371782:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10171.551709] LustreError: 371782:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10171.566915] LustreError: 371782:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10171.574391] LustreError: 371782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10192.784105] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10195.813941] Key type lgssc unregistered [10196.097216] LNet: 372785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10196.109100] LNetError: 372785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10196.133553] LNet: Removed LNI 192.168.203.139@tcp [10197.122736] Key type .llcrypt unregistered [10197.126638] Key type ._llcrypt unregistered [10218.596899] Key type ._llcrypt registered [10218.608671] Key type .llcrypt registered [10218.787491] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10233.544649] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10234.609273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10234.705508] alg: No test for adler32 (adler32-zlib) [10235.802649] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [10236.156755] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10237.879198] Key type lgssc registered [10239.009577] Lustre: Echo OBD driver; http://www.lustre.org/ [10278.459126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10278.491647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.789177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10279.855438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10279.991284] Lustre: lustre-MDT0000: new disk, initializing [10280.115779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10280.137907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10284.648561] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10296.707118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10296.822965] Lustre: 375971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10296.867461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10296.870049] Lustre: Skipped 1 previous similar message [10296.965667] Lustre: lustre-MDT0001: new disk, initializing [10297.044600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10297.083028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10297.099970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10302.099228] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10312.893785] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10318.696284] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10326.774945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10327.166597] Lustre: lustre-OST0000: new disk, initializing [10327.170268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10327.175881] Lustre: 377942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10327.296475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10332.779902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10332.787489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10332.940756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10334.491371] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10345.091145] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10350.362255] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10350.610840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10355.772758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10356.031148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10358.755793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10358.761897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.765996] LustreError: Skipped 1 previous similar message [10358.797444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10358.825227] Lustre: Skipped 1 previous similar message [10363.874319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.881826] Lustre: Skipped 1 previous similar message [10364.284937] Lustre: server umount lustre-OST0000 complete [10368.997299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10369.002548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10369.021718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10369.051391] Lustre: Skipped 2 previous similar messages [10374.113445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10374.123392] Lustre: Skipped 2 previous similar messages [10374.972176] Lustre: server umount lustre-MDT0000 complete [10379.332178] LustreError: 375966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110470 with bad export cookie 5966026499261517561 [10379.343865] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.351307] LustreError: 375966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10379.700550] Lustre: server umount lustre-MDT0001 complete [10388.903786] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10390.754956] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:54:41 (1790110481) [10397.455892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10397.868053] LustreError: 380400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.890688] LustreError: 380400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10397.962627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10402.787388] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10403.298582] LustreError: 380401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.420463] LustreError: 380400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.263798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.657803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10415.866737] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10423.187155] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10428.255327] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10434.349317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.679361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10440.264519] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10447.490879] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10453.350745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10453.558738] Lustre: lustre-OST0001: new disk, initializing [10453.562820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10453.568938] Lustre: 383236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10453.656841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10458.716251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10458.731585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10458.879079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10460.225715] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10468.474336] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10474.470932] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10474.471823] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10474.489707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10474.495839] Lustre: Skipped 1 previous similar message [10477.857337] Lustre: server umount lustre-OST0001 complete [10479.602632] LustreError: 382222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10479.625834] LustreError: 382222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10484.724085] LustreError: 382220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.740705] LustreError: 382220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10493.412632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10493.429073] LustreError: Skipped 1 previous similar message [10493.443331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.471335] Lustre: Skipped 1 previous similar message [10493.486506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10493.507927] Lustre: Skipped 1 previous similar message [10494.945463] LustreError: 382222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.972675] LustreError: 382222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10499.612959] Lustre: server umount lustre-OST0000 complete [10503.648433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10503.668458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.683260] Lustre: Skipped 1 previous similar message [10509.480604] Lustre: server umount lustre-MDT0000 complete [10512.814836] LustreError: 380382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110604 with bad export cookie 5966026499261518443 [10512.817974] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.823728] LustreError: 380382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10527.199146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10527.449652] Lustre: server umount lustre-MDT0001 complete [10536.927933] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10539.213332] Key type lgssc unregistered [10539.460294] LNet: 385224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10539.469634] LNetError: 385224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10539.485242] LNet: Removed LNI 192.168.203.139@tcp [10540.129559] Key type .llcrypt unregistered [10540.132139] Key type ._llcrypt unregistered [10586.148396] Key type ._llcrypt registered [10586.151308] Key type .llcrypt registered [10586.264534] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10587.537759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10587.634821] alg: No test for adler32 (adler32-zlib) [10588.737377] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [10589.044576] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10590.753687] Key type lgssc registered [10591.958933] Lustre: Echo OBD driver; http://www.lustre.org/ [10598.112047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10598.129181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.346871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10599.378509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10599.492577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10602.727909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10609.348288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10609.459155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10609.486927] Lustre: 388037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10609.716670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10613.449807] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10619.770276] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10624.671685] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10626.528657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10626.545639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10626.558838] Lustre: Skipped 1 previous similar message [10631.652419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10631.659948] Lustre: Skipped 1 previous similar message [10632.571984] Lustre: server umount lustre-MDT0000 complete [10635.721076] LustreError: 388778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110727 with bad export cookie 4110419300541977201 [10635.722322] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10635.735598] LustreError: 388778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10636.114421] Lustre: server umount lustre-MDT0001 complete [10643.958378] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:58:55 (1790110735) [10649.187109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.723279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10653.374990] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10655.204259] LustreError: 390549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.219948] LustreError: 390549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10659.612883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.890418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10663.236707] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10669.178186] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10673.733613] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10679.566384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10679.735821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10679.930703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10685.482965] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10692.231969] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10699.797124] Lustre: Permanently deactivating lustre-OST0000 [10699.813845] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10709.026177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10709.954924] LustreError: 393580:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10709.967861] LustreError: 393580:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10709.973884] LustreError: 393580:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10712.756964] LustreError: 393629:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10712.768616] LustreError: 393629:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10720.473339] Lustre: server umount lustre-OST0000 complete [10722.622451] LustreError: 393878:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10722.628576] LustreError: 393878:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10726.368385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10726.376605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.387076] Lustre: Skipped 1 previous similar message [10726.392329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10727.911107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.023803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10733.034091] Lustre: Skipped 3 previous similar messages [10734.122087] Lustre: server umount lustre-MDT0000 complete [10737.366802] LustreError: 390528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110828 with bad export cookie 4110419300541977782 [10737.376771] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10737.382176] LustreError: 390528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10746.968105] Lustre: server umount lustre-MDT0001 complete [10754.174549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.021636] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10760.232560] LustreError: 395035:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10764.466643] Lustre: 395374:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [10766.258953] Lustre: server umount MGS complete [10775.159131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.511859] LustreError: 396109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.530823] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10775.534520] Lustre: Skipped 1 previous similar message [10775.569732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10778.979243] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10780.643050] LustreError: 396110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.402802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.653464] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10789.314542] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10795.077169] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10799.174595] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10800.538867] Lustre: Permanently reactivating lustre-OST0000 [10800.555494] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10800.559552] Lustre: Skipped 2 previous similar messages [10828.767203] LustreError: 397939:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10834.002504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.280216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10834.283746] Lustre: Skipped 1 previous similar message [10838.601434] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10844.210433] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10847.713938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10847.718383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10847.728178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.742654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10849.697966] Lustre: server umount lustre-OST0000 complete [10853.857707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.874679] Lustre: Skipped 2 previous similar messages [10858.758532] Lustre: server umount lustre-MDT0000 complete [10858.979283] LustreError: 396109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10859.000789] LustreError: 396109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10861.789100] LustreError: 396090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110953 with bad export cookie 4110419300541978832 [10861.797628] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10861.799019] LustreError: 396090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10862.086364] Lustre: server umount lustre-MDT0001 complete [10870.434983] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10872.483073] Key type lgssc unregistered [10872.711336] LNet: 400614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10872.720455] LNetError: 400614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10872.737326] LNet: Removed LNI 192.168.203.139@tcp [10873.368892] Key type .llcrypt unregistered [10873.373258] Key type ._llcrypt unregistered [10887.511576] Key type ._llcrypt registered [10887.515341] Key type .llcrypt registered [10887.614695] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10898.301621] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10899.212547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10899.257571] alg: No test for adler32 (adler32-zlib) [10900.251755] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [10900.419790] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10902.047254] Key type lgssc registered [10902.799040] Lustre: Echo OBD driver; http://www.lustre.org/ [10929.335619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10929.355876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.617171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10930.639223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10930.696575] Lustre: lustre-MDT0000: new disk, initializing [10930.748852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10930.764592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10933.903910] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10941.945249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.022072] Lustre: 403764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10942.070214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10942.076900] Lustre: Skipped 1 previous similar message [10942.143137] Lustre: lustre-MDT0001: new disk, initializing [10942.210369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10942.250565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10942.260240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10945.095199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10951.285986] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10955.056550] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10960.358345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10960.598703] Lustre: lustre-OST0000: new disk, initializing [10960.602912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10960.606703] Lustre: 405732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10960.651910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10965.278639] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10969.120300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10969.129734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10969.213137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10972.224265] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10975.510865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10975.688604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10978.590480] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10978.736086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10984.421245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10984.425875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10984.433399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10984.445474] Lustre: Skipped 1 previous similar message [10986.169228] Lustre: server umount lustre-OST0000 complete [10989.538470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.553579] Lustre: Skipped 1 previous similar message [10989.562961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10989.573247] Lustre: Skipped 1 previous similar message [10994.663169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10994.668183] Lustre: Skipped 1 previous similar message [10994.829713] Lustre: server umount lustre-MDT0000 complete [10997.460325] LustreError: 404560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111089 with bad export cookie 17397257018951373458 [10997.461989] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10997.470564] LustreError: 404560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10997.701803] Lustre: server umount lustre-MDT0001 complete [11003.993040] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11005.059630] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:04:56 (1790111096) [11010.119822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.464392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11013.326140] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11015.650642] LustreError: 408190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.664490] LustreError: 408190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11019.394609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.633147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11022.564652] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11027.760207] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11031.799476] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11036.349464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11036.574394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11041.207318] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11046.438889] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11047.789216] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11059.188274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11065.312275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11065.317921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.325531] Lustre: Skipped 1 previous similar message [11065.329778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11066.338591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.538699] Lustre: server umount lustre-OST0000 complete [11070.945930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11070.952969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.960319] Lustre: Skipped 1 previous similar message [11070.966190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11070.970586] Lustre: Skipped 2 previous similar messages [11081.696997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11081.703216] Lustre: Skipped 5 previous similar messages [11085.279132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11085.397685] Lustre: server umount lustre-MDT0000 complete [11086.816743] LustreError: 408191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.834836] LustreError: 408191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11087.902395] LustreError: 408170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111179 with bad export cookie 17397257018951374340 [11087.903606] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11087.908431] LustreError: 408170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11088.100354] Lustre: server umount lustre-MDT0001 complete [11095.743156] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11097.830583] Key type lgssc unregistered [11098.017857] LNet: 411931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11098.022765] LNetError: 411931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11098.036185] LNet: Removed LNI 192.168.203.139@tcp [11098.601280] Key type .llcrypt unregistered [11098.603489] Key type ._llcrypt unregistered [11112.843140] Key type ._llcrypt registered [11112.845200] Key type .llcrypt registered [11112.933297] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11113.124335] Key type .llcrypt unregistered [11113.126365] Key type ._llcrypt unregistered [11125.374917] Key type ._llcrypt registered [11125.377510] Key type .llcrypt registered [11125.431571] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11126.113025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11126.129545] alg: No test for adler32 (adler32-zlib) [11127.073460] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11127.248214] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11128.879341] Key type lgssc registered [11129.704143] Lustre: Echo OBD driver; http://www.lustre.org/ [11135.203056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11135.221162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.527314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.509736] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11144.899528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11145.139461] LustreError: 414134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.151830] LustreError: 414134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11145.156146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11147.237053] LustreError: 414133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11147.247487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11147.250406] LustreError: 414133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11148.671304] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11152.354756] LustreError: 414134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.373074] LustreError: 414134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11153.797481] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11157.479119] LustreError: 414133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.489291] LustreError: 414133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11158.009052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.264621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11160.669309] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11172.832081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11172.838815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.852735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.859176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11173.860397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.868702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.876855] Lustre: Skipped 1 previous similar message [11175.864831] Lustre: server umount lustre-OST0000 complete [11178.977309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11178.978745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11178.990256] Lustre: Skipped 1 previous similar message [11179.000941] Lustre: Skipped 1 previous similar message [11184.099189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11184.105802] Lustre: Skipped 1 previous similar message [11184.390840] Lustre: server umount lustre-MDT0000 complete [11186.709374] LustreError: 414113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111278 with bad export cookie 6356145213872123098 [11186.714030] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11186.714486] LustreError: 414113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11186.902918] Lustre: server umount lustre-MDT0001 complete [11192.899495] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11194.440368] Key type lgssc unregistered [11194.613496] LNet: 417281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11194.619155] LNetError: 417281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11194.628342] LNet: Removed LNI 192.168.203.139@tcp [11195.059839] Key type .llcrypt unregistered [11195.061191] Key type ._llcrypt unregistered [11210.097048] Key type ._llcrypt registered [11210.100595] Key type .llcrypt registered [11210.198624] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11210.785061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11210.883585] alg: No test for adler32 (adler32-zlib) [11211.837525] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11212.072547] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11213.735064] Key type lgssc registered [11214.693789] Lustre: Echo OBD driver; http://www.lustre.org/ [11220.386803] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11220.401986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.935894] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11241.055828] LustreError: 418663:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [11267.679267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11271.721630] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11276.987133] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11281.580522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.889938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11281.928780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11284.746300] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11288.037517] LustreError: 418690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.057467] LustreError: 418690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11290.858849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.128118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11294.426346] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11300.439141] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11304.823288] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11311.607864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11311.608636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.616402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11311.628451] Lustre: Skipped 1 previous similar message [11311.647840] Lustre: Skipped 1 previous similar message [11315.636652] Lustre: server umount lustre-OST0000 complete [11318.758368] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11318.763606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.777206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11320.355512] Lustre: server umount lustre-MDT0000 complete [11321.829200] LustreError: 419706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.829356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.839095] LustreError: 418670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111414 with bad export cookie 13468682516036188901 [11322.842382] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11322.847692] LustreError: 418670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11323.064351] Lustre: server umount lustre-MDT0001 complete [11330.667143] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11334.965715] Key type lgssc unregistered [11335.404817] LNet: 422230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11335.421369] LNetError: 422230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11335.443755] LNet: Removed LNI 192.168.203.139@tcp [11337.105840] Key type .llcrypt unregistered [11337.108298] Key type ._llcrypt unregistered [11370.665637] Key type ._llcrypt registered [11370.673393] Key type .llcrypt registered [11370.780573] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11372.240568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11372.308693] alg: No test for adler32 (adler32-zlib) [11373.471898] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11373.770408] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11375.463223] Key type lgssc registered [11376.880687] Lustre: Echo OBD driver; http://www.lustre.org/ [11383.979664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11384.002974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.454039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.560103] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11394.529091] LustreError: 423663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.551123] LustreError: 423663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11399.652558] LustreError: 423664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.490427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11402.131657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11409.047910] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11418.592843] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11424.829752] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11430.948597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11431.258498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11436.607786] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11442.950766] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11454.849231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11463.653776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11463.662778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.677883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11465.030596] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [11465.188407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11465.207498] Lustre: Skipped 1 previous similar message [11468.923703] Lustre: server umount lustre-MDT0000 complete [11470.121650] LustreError: 425839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.050813] LustreError: 423644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.39@tcp arrived at 1790111562 with bad export cookie 10747059504048755930 [11471.076126] LustreError: 423644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11472.827214] LustreError: 423645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111564 with bad export cookie 10747059504048755272 [11472.835214] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.847642] LustreError: 423645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11475.426427] LustreError: 423660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.426763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.447841] LustreError: 423660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11475.479396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11475.483146] Lustre: Skipped 3 previous similar messages [11479.034531] Lustre: server umount lustre-OST0000 complete [11482.594980] LustreError: 423664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.622749] LustreError: 423664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11485.994887] Lustre: server umount lustre-MDT0001 complete [11497.196927] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11500.126847] Key type lgssc unregistered [11500.497053] LNet: 427370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11500.509536] LNetError: 427370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11500.523954] LNet: Removed LNI 192.168.203.139@tcp [11501.409767] Key type .llcrypt unregistered [11501.412321] Key type ._llcrypt unregistered [11525.520295] Key type ._llcrypt registered [11525.524685] Key type .llcrypt registered [11525.598980] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11526.762395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11526.844908] alg: No test for adler32 (adler32-zlib) [11527.899570] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11528.149859] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11529.889443] Key type lgssc registered [11531.118353] Lustre: Echo OBD driver; http://www.lustre.org/ [11539.381777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11539.399363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.899071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11545.004206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11551.203489] LustreError: 428804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.225238] LustreError: 428804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11554.108269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.499207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11559.192691] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11566.709836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11572.850335] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11579.704972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.142791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11586.175257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11586.285454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11593.942978] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11600.153896] Lustre: server umount lustre-MDT0001 complete [11600.721082] LustreError: 428800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.868838] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11600.876959] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.894620] LustreError: 428804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.898451] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.918328] LustreError: 428804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11607.009555] LustreError: 428823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.012685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11607.037466] LustreError: 428823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11607.054956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11612.129278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11612.130311] LustreError: 428805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11613.126862] Lustre: server umount lustre-OST0000 complete [11617.096809] Lustre: server umount lustre-MDT0000 complete [11631.274341] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11634.368420] Key type lgssc unregistered [11634.670237] LNet: 432488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11634.684151] LNetError: 432488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.698921] LNet: Removed LNI 192.168.203.139@tcp [11635.578591] Key type .llcrypt unregistered [11635.581102] Key type ._llcrypt unregistered [11661.129597] Key type ._llcrypt registered [11661.133177] Key type .llcrypt registered [11661.279996] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11676.362350] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11677.348125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11677.491904] alg: No test for adler32 (adler32-zlib) [11678.652159] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11679.030318] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11680.743237] Key type lgssc registered [11682.016399] Lustre: Echo OBD driver; http://www.lustre.org/ [11726.894197] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11743.599478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11743.619310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.953490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11745.013060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11745.160702] Lustre: lustre-MDT0000: new disk, initializing [11745.334491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11745.359445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11750.522634] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11767.085550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11767.545760] Lustre: lustre-OST0000: new disk, initializing [11767.559699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11767.571077] Lustre: Skipped 1 previous similar message [11767.575227] Lustre: 437543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11767.732892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11774.127694] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11776.595765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11776.606994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11776.640542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11784.113633] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11800.738548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.859200] Lustre: 436523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11800.977125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11801.102853] Lustre: lustre-MDT0001: new disk, initializing [11801.228996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11801.287546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11801.320046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11806.642875] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11817.878416] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11818.123775] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11822.837766] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11823.110324] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11829.065273] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [11829.729890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11829.732997] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11829.746295] Lustre: Skipped 1 previous similar message [11834.186990] Lustre: server umount lustre-MDT0001 complete [11839.289694] LustreError: 436528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11839.314219] LustreError: 436528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11839.972985] LustreError: 436529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11839.986084] LustreError: 436529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11845.091898] LustreError: 436533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.118595] LustreError: 436533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11850.221762] LustreError: 439406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.244552] LustreError: 439406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11853.292103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11853.298291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.320367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11853.333584] Lustre: Skipped 1 previous similar message [11855.335304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11855.336162] LustreError: 436533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.760290] Lustre: server umount lustre-OST0000 complete [11869.299836] Lustre: server umount lustre-MDT0000 complete [11884.735625] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11887.200829] Key type lgssc unregistered [11887.455708] LNet: 441297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11887.467206] LNetError: 441297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11887.489682] LNet: Removed LNI 192.168.203.139@tcp [11888.316694] Key type .llcrypt unregistered [11888.318844] Key type ._llcrypt unregistered [11925.991192] Key type ._llcrypt registered [11925.997184] Key type .llcrypt registered [11926.137256] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11926.430464] Key type .llcrypt unregistered [11926.435715] Key type ._llcrypt unregistered [11943.671346] Key type ._llcrypt registered [11943.673737] Key type .llcrypt registered [11943.777930] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11945.140477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11945.183262] alg: No test for adler32 (adler32-zlib) [11946.190994] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [11946.429764] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11948.087415] Key type lgssc registered [11949.530808] Lustre: Echo OBD driver; http://www.lustre.org/ [11978.007987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11978.058052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11979.324692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11979.369344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11979.446394] Lustre: lustre-MDT0000: new disk, initializing [11979.580047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11979.605787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11984.033937] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11994.443528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.545329] Lustre: 444685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11994.597956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11994.601887] Lustre: Skipped 1 previous similar message [11994.662796] Lustre: lustre-MDT0001: new disk, initializing [11994.746602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11994.779755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11994.801341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11999.171518] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12008.559736] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12013.946108] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12020.276269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12020.543168] Lustre: lustre-OST0000: new disk, initializing [12020.551639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12020.565841] Lustre: 446653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12020.681441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12026.563647] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12027.449180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12027.460691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12027.596493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12035.400833] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12043.234784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12043.237246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12043.249918] Lustre: Skipped 1 previous similar message [12047.610867] Lustre: server umount lustre-MDT0000 complete [12051.160437] LustreError: 444678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790112142 with bad export cookie 2185646823752386697 [12051.174171] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12051.174592] LustreError: 444678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12051.560749] Lustre: server umount lustre-MDT0001 complete [12064.675646] Lustre: server umount lustre-OST0000 complete [12073.950535] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:22:44 (1790112164) [12081.759366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.095622] LustreError: 448685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.113053] LustreError: 448685:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12082.152185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12086.944207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12087.271902] LustreError: 448686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.389985] LustreError: 448685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.187861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12095.517391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12099.797289] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12107.261460] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12112.975982] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12132.918754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12133.331354] Lustre: Found index 0 for lustre-OST0000, updating log [12133.574493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12140.028976] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12148.248318] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12153.021070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12161.508332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12161.523849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.534802] Lustre: Skipped 1 previous similar message [12161.544933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12161.558891] Lustre: Skipped 2 previous similar messages [12162.594796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12162.603515] Lustre: Skipped 1 previous similar message [12165.478439] Lustre: server umount lustre-OST0000 complete [12182.496979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12182.503746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12182.520527] Lustre: Skipped 1 previous similar message [12182.526830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12182.538689] Lustre: Skipped 1 previous similar message [12187.808441] Lustre: server umount lustre-MDT0000 complete [12189.156304] LustreError: 449926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.191203] LustreError: 449926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12191.802559] LustreError: 448666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790112283 with bad export cookie 2185646823752388090 [12191.806924] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12191.823820] LustreError: 448666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12192.219393] Lustre: server umount lustre-MDT0001 complete [12196.320840] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [12201.014409] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:24:51 (1790112291) [12822.193806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.620493] LustreError: 455094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.646397] LustreError: 455094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12822.685828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12827.802628] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12828.136025] LustreError: 455095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.255138] LustreError: 455094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12837.062221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.537951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12841.782620] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12849.052768] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12854.300197] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12873.647596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12873.945695] Lustre: Found index 0 for lustre-OST0000, updating log [12874.200948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12875.231363] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [12880.510551] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12887.354265] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12889.183917] Lustre: server umount lustre-OST0000 complete [12893.411359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12893.414210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12893.422937] Lustre: Skipped 2 previous similar messages [12893.430125] Lustre: Skipped 3 previous similar messages [12894.432772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12894.441889] Lustre: Skipped 1 previous similar message [12895.457993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12895.466163] Lustre: Skipped 1 previous similar message [12897.511242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12897.521035] Lustre: Skipped 2 previous similar messages [12898.713268] Lustre: server umount lustre-MDT0000 complete [12899.616858] LustreError: 455090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.625964] LustreError: 455090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12902.400133] LustreError: 455074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790112993 with bad export cookie 2185646823752389175 [12902.405307] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12902.420034] LustreError: 455074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12902.873906] Lustre: server umount lustre-MDT0001 complete [12934.634688] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [12934.914800] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12945.366663] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12979.859357] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:37:50 (1790113070) [12994.630876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12997.242382] Key type lgssc unregistered [12997.512841] LNet: 461707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12997.520327] LNetError: 461707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13084.584434] LNet: Removed LNI 192.168.203.139@tcp [13085.456480] Key type .llcrypt unregistered [13085.459805] Key type ._llcrypt unregistered [13101.098522] Key type ._llcrypt registered [13101.100687] Key type .llcrypt registered [13101.212469] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13101.920614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13102.038261] alg: No test for adler32 (adler32-zlib) [13103.026614] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [13103.213966] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13104.872197] Key type lgssc registered [13105.959883] Lustre: Echo OBD driver; http://www.lustre.org/ [13109.677045] LNet: Added LNI 192.168.203.139@tcp100 [8/256/0/180] [13110.471359] LNet: Added LNI 192.168.203.139@tcp101 [8/256/0/180] [13111.301295] LNet: Added LNI 192.168.203.139@tcp102 [8/256/0/180] [13112.159420] LNet: Added LNI 192.168.203.139@tcp103 [8/256/0/180] [13112.948564] LNet: Added LNI 192.168.203.139@tcp104 [8/256/0/180] [13113.682794] LNet: Added LNI 192.168.203.139@tcp105 [8/256/0/180] [13114.486670] LNet: Added LNI 192.168.203.139@tcp106 [8/256/0/180] [13115.288846] LNet: Added LNI 192.168.203.139@tcp107 [8/256/0/180] [13115.989540] LNet: Added LNI 192.168.203.139@tcp108 [8/256/0/180] [13116.822769] LNet: Added LNI 192.168.203.139@tcp109 [8/256/0/180] [13117.667877] LNet: Added LNI 192.168.203.139@tcp110 [8/256/0/180] [13118.575738] LNet: Added LNI 192.168.203.139@tcp111 [8/256/0/180] [13119.446825] LNet: Added LNI 192.168.203.139@tcp112 [8/256/0/180] [13120.341589] LNet: Added LNI 192.168.203.139@tcp113 [8/256/0/180] [13121.248402] LNet: Added LNI 192.168.203.139@tcp114 [8/256/0/180] [13122.065200] LNet: Added LNI 192.168.203.139@tcp115 [8/256/0/180] [13123.028918] LNet: Added LNI 192.168.203.139@tcp116 [8/256/0/180] [13123.901508] LNet: Added LNI 192.168.203.139@tcp117 [8/256/0/180] [13124.848985] LNet: Added LNI 192.168.203.139@tcp118 [8/256/0/180] [13125.706740] LNet: Added LNI 192.168.203.139@tcp119 [8/256/0/180] [13126.655750] LNet: Added LNI 192.168.203.139@tcp120 [8/256/0/180] [13127.571690] LNet: Added LNI 192.168.203.139@tcp121 [8/256/0/180] [13128.579636] LNet: Added LNI 192.168.203.139@tcp122 [8/256/0/180] [13129.457867] LNet: Added LNI 192.168.203.139@tcp123 [8/256/0/180] [13130.426696] LNet: Added LNI 192.168.203.139@tcp124 [8/256/0/180] [13131.252619] LNet: Added LNI 192.168.203.139@tcp125 [8/256/0/180] [13132.086508] LNet: Added LNI 192.168.203.139@tcp126 [8/256/0/180] [13132.854740] LNet: Added LNI 192.168.203.139@tcp127 [8/256/0/180] [13133.681096] LNet: Added LNI 192.168.203.139@tcp128 [8/256/0/180] [13134.720904] LNet: Added LNI 192.168.203.139@tcp129 [8/256/0/180] [13135.605276] LNet: Added LNI 192.168.203.139@tcp130 [8/256/0/180] [13136.461588] LNet: Added LNI 192.168.203.139@tcp131 [8/256/0/180] [13137.293405] LNet: Added LNI 192.168.203.139@tcp132 [8/256/0/180] [13138.019756] LNet: Added LNI 192.168.203.139@tcp133 [8/256/0/180] [13138.792426] LNet: Added LNI 192.168.203.139@tcp134 [8/256/0/180] [13139.519958] LNet: Added LNI 192.168.203.139@tcp135 [8/256/0/180] [13145.353953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13145.403069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.490609] Lustre: 464751:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [13146.531153] LNetError: 464774:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13146.540940] LNetError: Protocol error connecting to 192.168.203.139@tcp118 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13146.549127] LNetError: Skipped 2 previous similar messages [13146.643129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13146.692265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13146.789806] Lustre: lustre-MDT0000: new disk, initializing [13146.863458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13146.877605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13147.295325] LNet: 4 local NIs in recovery (showing 4): 192.168.203.139@tcp124, 192.168.203.139@tcp125, 192.168.203.139@tcp128, 192.168.203.139@tcp129 [13147.298276] LNetError: 464771:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13147.341501] LNetError: 464771:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [13147.349219] LNetError: Protocol error connecting to 192.168.203.139@tcp102 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13147.370889] LNetError: Skipped 4 previous similar messages [13151.120953] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13160.549265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13160.645701] Lustre: 464768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13160.696899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13160.700699] Lustre: Skipped 1 previous similar message [13160.782633] Lustre: lustre-MDT0001: new disk, initializing [13160.809459] Lustre: 465570:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [13160.819109] Lustre: 465570:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13160.942966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13160.975301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13160.991579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13164.599756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13172.006873] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13176.500849] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13176.930926] LNetError: 464772:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13176.936846] LNetError: Protocol error connecting to 192.168.203.139@tcp123 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13178.983857] LNetError: 464773:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13178.999738] LNetError: 464773:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13179.008079] LNetError: Protocol error connecting to 192.168.203.139@tcp126 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13179.023742] LNetError: Skipped 1 previous similar message [13182.209905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13182.504315] Lustre: lustre-OST0000: new disk, initializing [13182.507697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13182.512810] Lustre: 466742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13182.595126] Lustre: 466747:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [13182.602136] Lustre: 466747:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13182.649556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13183.073216] LNetError: 462201:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13183.086621] LNetError: 462201:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13183.101433] LNetError: Protocol error connecting to 192.168.203.139@tcp127 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13183.113783] LNetError: Skipped 1 previous similar message [13187.970925] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13190.714836] Lustre: 467218:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.139@tcp: can't add NID 192.168.203.139@tcp130: rc = -75 [13190.720966] Lustre: 467218:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13190.771982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13190.779254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13190.884342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13191.264186] LNetError: 462200:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13191.277138] LNetError: 462200:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [13191.285163] LNetError: Protocol error connecting to 192.168.203.139@tcp126 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13191.291140] LNetError: Skipped 3 previous similar messages [13196.005375] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13206.506484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13206.514771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13206.543867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13206.565140] Lustre: Skipped 1 previous similar message [13208.672681] LNetError: 464774:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13208.692148] LNetError: 464774:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13208.704695] LNetError: Protocol error connecting to 192.168.203.139@tcp127 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13208.712596] LNetError: Skipped 7 previous similar messages [13211.623119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13211.637619] Lustre: Skipped 1 previous similar message [13216.224215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13216.514047] Lustre: server umount lustre-OST0000 complete [13221.857081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13221.895895] Lustre: Skipped 1 previous similar message [13221.909337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13226.816044] Lustre: server umount lustre-MDT0000 complete [13230.828489] LustreError: 465569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790113322 with bad export cookie 9468190866896586701 [13230.833364] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13230.842288] LustreError: 465569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13231.139771] Lustre: server umount lustre-MDT0001 complete [13241.453879] LNetError: 462202:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.139 [13241.463419] LNetError: 462202:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13241.480921] LNetError: Protocol error connecting to 192.168.203.139@tcp118 at host 192.168.203.139:988: is it running a compatible version of Lustre? [13241.497476] LNetError: Skipped 8 previous similar messages [13241.558898] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13244.802189] Key type lgssc unregistered [13245.134382] LNet: 468654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13245.142963] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13245.150984] LNet: Removed LNI 192.168.203.139@tcp [13246.175319] LNet: Removed LNI 192.168.203.139@tcp100 [13246.179833] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13246.195904] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13246.212969] LNet: Removed LNI 192.168.203.139@tcp101 [13247.263621] LNet: Removed LNI 192.168.203.139@tcp102 [13247.267555] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13247.277792] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13247.290284] LNet: Removed LNI 192.168.203.139@tcp103 [13248.351203] LNet: Removed LNI 192.168.203.139@tcp104 [13249.375163] LNet: Removed LNI 192.168.203.139@tcp105 [13249.385928] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13249.400461] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13249.409134] LNet: Removed LNI 192.168.203.139@tcp106 [13250.463253] LNet: Removed LNI 192.168.203.139@tcp107 [13251.487191] LNet: Removed LNI 192.168.203.139@tcp108 [13252.575186] LNet: Removed LNI 192.168.203.139@tcp109 [13253.599371] LNet: Removed LNI 192.168.203.139@tcp110 [13253.608266] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13253.616291] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13253.629048] LNet: Removed LNI 192.168.203.139@tcp111 [13254.687257] LNet: Removed LNI 192.168.203.139@tcp112 [13255.713383] LNet: Removed LNI 192.168.203.139@tcp113 [13256.735283] LNet: Removed LNI 192.168.203.139@tcp114 [13257.759348] LNet: Removed LNI 192.168.203.139@tcp115 [13258.783246] LNet: Removed LNI 192.168.203.139@tcp116 [13259.807465] LNet: Removed LNI 192.168.203.139@tcp117 [13260.832104] LNet: Removed LNI 192.168.203.139@tcp118 [13261.855220] LNet: Removed LNI 192.168.203.139@tcp119 [13261.857655] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13261.868863] LNetError: 468654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13261.882892] LNet: Removed LNI 192.168.203.139@tcp120 [13262.943209] LNet: Removed LNI 192.168.203.139@tcp121 [13263.967176] LNet: Removed LNI 192.168.203.139@tcp122 [13264.991223] LNet: Removed LNI 192.168.203.139@tcp123 [13266.016789] LNet: Removed LNI 192.168.203.139@tcp124 [13267.039184] LNet: Removed LNI 192.168.203.139@tcp125 [13268.063155] LNet: Removed LNI 192.168.203.139@tcp126 [13269.087323] LNet: Removed LNI 192.168.203.139@tcp127 [13270.111214] LNet: Removed LNI 192.168.203.139@tcp128 [13271.135235] LNet: Removed LNI 192.168.203.139@tcp129 [13271.142662] LNet: Removed LNI 192.168.203.139@tcp130 [13271.147679] LNet: Removed LNI 192.168.203.139@tcp131 [13271.159329] LNet: Removed LNI 192.168.203.139@tcp132 [13271.161344] LNet: Removed LNI 192.168.203.139@tcp133 [13271.164334] LNet: Removed LNI 192.168.203.139@tcp134 [13271.186220] LNet: Removed LNI 192.168.203.139@tcp135 [13271.944777] Key type .llcrypt unregistered [13271.951056] Key type ._llcrypt unregistered [13298.675597] Key type ._llcrypt registered [13298.678200] Key type .llcrypt registered [13298.777562] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [13312.538868] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13313.717875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13313.756354] alg: No test for adler32 (adler32-zlib) [13314.761671] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [13314.950700] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13316.617745] Key type lgssc registered [13317.722558] Lustre: Echo OBD driver; http://www.lustre.org/ [13360.222613] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:44:10 (1790113450) [13379.232440] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13381.966589] Key type lgssc unregistered [13382.301151] LNet: 472757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13382.309172] LNetError: 472757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13382.328161] LNet: Removed LNI 192.168.203.139@tcp [13383.310546] Key type .llcrypt unregistered [13383.314462] Key type ._llcrypt unregistered [13399.790352] Key type ._llcrypt registered [13399.792083] Key type .llcrypt registered [13399.881495] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13400.561802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13400.587979] alg: No test for adler32 (adler32-zlib) [13401.655513] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [13401.902386] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13403.567254] Key type lgssc registered [13404.847814] Lustre: Echo OBD driver; http://www.lustre.org/ [13413.644031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13413.675880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13414.999475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13415.024023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13415.115483] Lustre: lustre-MDT0000: new disk, initializing [13415.200678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13415.223384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13419.683389] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13429.029047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13429.142716] Lustre: 473987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13429.223210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13429.228611] Lustre: Skipped 1 previous similar message [13429.320834] Lustre: lustre-MDT0001: new disk, initializing [13429.460328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13429.502224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13429.515795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13433.047660] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13440.719764] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13445.026440] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13450.585555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13450.793535] Lustre: lustre-OST0000: new disk, initializing [13450.799763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13450.807219] Lustre: 475953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13450.878608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13456.144376] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13457.969425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13457.987420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13458.109432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13464.401235] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13467.582792] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing lnet_if_list [13472.371553] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13472.413123] LNet: Added LNI 192.168.203.139@tcp10 [8/256/0/180] [13477.622724] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13477.656127] LNet: Added LNI 192.168.203.139@tcp11 [8/256/0/180] [13481.480238] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13481.505022] LNet: Added LNI 192.168.203.139@tcp12 [8/256/0/180] [13485.739455] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13485.767955] LNet: Added LNI 192.168.203.139@tcp13 [8/256/0/180] [13489.660370] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13489.706051] LNet: Added LNI 192.168.203.139@tcp14 [8/256/0/180] [13493.364227] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13493.387175] LNet: Added LNI 192.168.203.139@tcp15 [8/256/0/180] [13497.853253] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13497.885716] LNet: Added LNI 192.168.203.139@tcp16 [8/256/0/180] [13502.245746] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13502.279473] LNet: Added LNI 192.168.203.139@tcp17 [8/256/0/180] [13507.217672] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13507.260905] LNet: Added LNI 192.168.203.139@tcp18 [8/256/0/180] [13512.171943] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13512.191323] LNet: Added LNI 192.168.203.139@tcp19 [8/256/0/180] [13516.879469] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13516.909558] LNet: Added LNI 192.168.203.139@tcp20 [8/256/0/180] [13521.309060] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13521.341784] LNet: Added LNI 192.168.203.139@tcp21 [8/256/0/180] [13526.198364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13526.237682] LNet: Added LNI 192.168.203.139@tcp22 [8/256/0/180] [13531.197168] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13531.222894] LNet: Added LNI 192.168.203.139@tcp23 [8/256/0/180] [13536.699662] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13536.738691] LNet: Added LNI 192.168.203.139@tcp24 [8/256/0/180] [13541.782964] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13541.802442] LNet: Added LNI 192.168.203.139@tcp25 [8/256/0/180] [13547.649396] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13547.668827] LNet: Added LNI 192.168.203.139@tcp26 [8/256/0/180] [13553.375281] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13553.408831] LNet: Added LNI 192.168.203.139@tcp27 [8/256/0/180] [13559.303600] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13559.333284] LNet: Added LNI 192.168.203.139@tcp28 [8/256/0/180] [13565.453465] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13565.474861] LNet: Added LNI 192.168.203.139@tcp29 [8/256/0/180] [13570.994846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13571.025633] LNet: Added LNI 192.168.203.139@tcp30 [8/256/0/180] [13576.152178] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13576.197731] LNet: Added LNI 192.168.203.139@tcp31 [8/256/0/180] [13581.703718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13581.734756] LNet: Added LNI 192.168.203.139@tcp32 [8/256/0/180] [13587.651160] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13587.678479] LNet: Added LNI 192.168.203.139@tcp33 [8/256/0/180] [13593.312985] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13593.337588] LNet: Added LNI 192.168.203.139@tcp34 [8/256/0/180] [13599.077087] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13599.102564] LNet: Added LNI 192.168.203.139@tcp35 [8/256/0/180] [13604.159477] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13604.196410] LNet: Added LNI 192.168.203.139@tcp36 [8/256/0/180] [13609.770057] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13609.793253] LNet: Added LNI 192.168.203.139@tcp37 [8/256/0/180] [13615.165206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13615.215083] LNet: Added LNI 192.168.203.139@tcp38 [8/256/0/180] [13621.344323] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13621.372821] LNet: Added LNI 192.168.203.139@tcp39 [8/256/0/180] [13628.329427] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13628.362411] LNet: Added LNI 192.168.203.139@tcp40 [8/256/0/180] [13635.018274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13635.046939] LNet: Added LNI 192.168.203.139@tcp41 [8/256/0/180] [13642.538235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13642.606819] LNet: Added LNI 192.168.203.139@tcp42 [8/256/0/180] [13652.073739] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13652.106337] LNet: Added LNI 192.168.203.139@tcp43 [8/256/0/180] [13662.101383] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13662.133878] LNet: Added LNI 192.168.203.139@tcp44 [8/256/0/180] [13670.390392] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13670.419825] LNet: Added LNI 192.168.203.139@tcp45 [8/256/0/180] [13679.834633] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13679.871027] LNet: Added LNI 192.168.203.139@tcp46 [8/256/0/180] [13688.726116] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13688.774460] LNet: Added LNI 192.168.203.139@tcp47 [8/256/0/180] [13697.697969] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13697.741658] LNet: Added LNI 192.168.203.139@tcp48 [8/256/0/180] [13705.827572] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13705.884110] LNet: Added LNI 192.168.203.139@tcp49 [8/256/0/180] [13716.299979] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13716.327615] LNet: Added LNI 192.168.203.139@tcp50 [8/256/0/180] [13736.928529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13736.936805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.945921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13740.006515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13740.009365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13742.724915] Lustre: server umount lustre-MDT0000 complete [13746.396907] LustreError: 473976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790113837 with bad export cookie 11459268718960279435 [13746.398050] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13746.404348] LustreError: 473976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13746.676688] Lustre: server umount lustre-MDT0001 complete [13765.600192] Lustre: 473248:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790113841/real 1790113841] req@0000000083f7bd80 x1877070046195968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790113857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13765.626993] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13765.642513] Lustre: Skipped 1 previous similar message [13766.623108] Lustre: 473249:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790113842/real 1790113842] req@000000006aa467d7 x1877070046196096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790113858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13766.764703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13770.726218] Lustre: 473251:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790113846/real 1790113846] req@0000000028d2e782 x1877070046196224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790113862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13772.777286] Lustre: 473250:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790113848/real 1790113848] req@0000000070f85180 x1877070046196480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790113864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13778.634525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13778.911340] Lustre: 473249:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790113854/real 1790113854] req@00000000045600c8 x1877070046196608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790113870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13792.604516] LustreError: 475947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.619939] LustreError: 475947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13792.658515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13794.719731] LustreError: 475948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13794.720979] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13796.477614] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13796.772220] LustreError: 475947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13801.954565] LustreError: 475948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13803.434389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13803.805658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13807.881457] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13811.193632] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13814.899503] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13820.600214] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13839.845477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13839.853273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13839.874817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13839.880051] Lustre: Skipped 2 previous similar messages [13841.888287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13841.893285] Lustre: Skipped 1 previous similar message [13847.009308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13847.016405] Lustre: Skipped 1 previous similar message [13854.175131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13854.332198] Lustre: server umount lustre-OST0000 complete [13860.322366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13860.332181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13860.348726] Lustre: Skipped 1 previous similar message [13860.356251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13860.363782] Lustre: Skipped 2 previous similar messages [13863.735420] Lustre: server umount lustre-MDT0000 complete [13867.492559] LustreError: 501247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13867.515244] LustreError: 501247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13867.687388] LustreError: 474781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790113959 with bad export cookie 11459268718960281997 [13867.698354] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13868.165942] Lustre: server umount lustre-MDT0001 complete [13882.050558] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [13885.923600] Key type lgssc unregistered [13886.512486] LNet: 502988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13886.534036] LNetError: 502988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13886.552182] LNet: Removed LNI 192.168.203.139@tcp [13886.557592] LNet: Removed LNI 192.168.203.139@tcp10 [13886.562032] LNet: Removed LNI 192.168.203.139@tcp11 [13886.564464] LNet: Removed LNI 192.168.203.139@tcp12 [13886.566541] LNet: Removed LNI 192.168.203.139@tcp13 [13886.578971] LNet: Removed LNI 192.168.203.139@tcp14 [13886.584152] LNet: Removed LNI 192.168.203.139@tcp15 [13886.595290] LNet: Removed LNI 192.168.203.139@tcp16 [13886.597740] LNet: Removed LNI 192.168.203.139@tcp17 [13886.603297] LNet: Removed LNI 192.168.203.139@tcp18 [13886.605197] LNet: Removed LNI 192.168.203.139@tcp19 [13886.615572] LNet: Removed LNI 192.168.203.139@tcp20 [13886.621390] LNet: Removed LNI 192.168.203.139@tcp21 [13886.624286] LNet: Removed LNI 192.168.203.139@tcp22 [13886.626341] LNet: Removed LNI 192.168.203.139@tcp23 [13886.629861] LNet: Removed LNI 192.168.203.139@tcp24 [13886.633285] LNet: Removed LNI 192.168.203.139@tcp25 [13886.637456] LNet: Removed LNI 192.168.203.139@tcp26 [13886.642812] LNet: Removed LNI 192.168.203.139@tcp27 [13886.650719] LNet: Removed LNI 192.168.203.139@tcp28 [13886.656129] LNet: Removed LNI 192.168.203.139@tcp29 [13886.659935] LNet: Removed LNI 192.168.203.139@tcp30 [13886.668795] LNet: Removed LNI 192.168.203.139@tcp31 [13886.670960] LNet: Removed LNI 192.168.203.139@tcp32 [13886.676310] LNet: Removed LNI 192.168.203.139@tcp33 [13886.680233] LNet: Removed LNI 192.168.203.139@tcp34 [13886.685434] LNet: Removed LNI 192.168.203.139@tcp35 [13886.690376] LNet: Removed LNI 192.168.203.139@tcp36 [13886.695159] LNet: Removed LNI 192.168.203.139@tcp37 [13886.702192] LNet: Removed LNI 192.168.203.139@tcp38 [13886.722132] LNet: Removed LNI 192.168.203.139@tcp39 [13886.724535] LNet: Removed LNI 192.168.203.139@tcp40 [13886.733498] LNet: Removed LNI 192.168.203.139@tcp41 [13886.737420] LNet: Removed LNI 192.168.203.139@tcp42 [13886.739386] LNet: Removed LNI 192.168.203.139@tcp43 [13886.745123] LNet: Removed LNI 192.168.203.139@tcp44 [13886.750324] LNet: Removed LNI 192.168.203.139@tcp45 [13886.758745] LNet: Removed LNI 192.168.203.139@tcp46 [13886.761807] LNet: Removed LNI 192.168.203.139@tcp47 [13886.763719] LNet: Removed LNI 192.168.203.139@tcp48 [13886.768614] LNet: Removed LNI 192.168.203.139@tcp49 [13886.794556] LNet: Removed LNI 192.168.203.139@tcp50 [13888.027884] Key type .llcrypt unregistered [13888.036313] Key type ._llcrypt unregistered [13908.857179] Key type ._llcrypt registered [13908.860403] Key type .llcrypt registered [13908.949368] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [13925.579268] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [13926.626428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13926.774955] alg: No test for adler32 (adler32-zlib) [13927.784958] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [13928.011044] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [13929.647353] Key type lgssc registered [13931.079547] Lustre: Echo OBD driver; http://www.lustre.org/ [13963.797460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13963.853530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13965.052723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13965.085530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13965.166661] Lustre: lustre-MDT0000: new disk, initializing [13965.240497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13965.271112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13969.466700] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13980.535825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.658114] Lustre: 506200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13980.694457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13980.698815] Lustre: Skipped 1 previous similar message [13980.747224] Lustre: lustre-MDT0001: new disk, initializing [13980.914258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13980.960967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13980.968655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13986.356922] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [13995.413441] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14001.317038] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14007.445810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14007.594391] Lustre: *** cfs_fail_loc=90f, val=0*** [14007.646388] Lustre: lustre-OST0000: new disk, initializing [14007.650396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14007.659643] Lustre: 508218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14007.769222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14012.757721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14017.100738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14017.114325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14017.234682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14019.882825] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14032.929411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14032.936152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14032.946904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14034.813327] Lustre: server umount lustre-OST0000 complete [14042.599055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14042.628941] LustreError: Skipped 1 previous similar message [14042.642569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14042.681557] Lustre: Skipped 1 previous similar message [14042.690429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14042.694215] Lustre: Skipped 2 previous similar messages [14048.736918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14048.746808] Lustre: Skipped 3 previous similar messages [14053.870410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14053.883784] Lustre: Skipped 1 previous similar message [14054.578830] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14056.858197] Lustre: server umount lustre-MDT0000 complete [14062.853487] LustreError: 506193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790114154 with bad export cookie 6363964258452956200 [14062.854615] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14062.866589] LustreError: 506193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14063.617815] Lustre: server umount lustre-MDT0001 complete [14081.912933] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14084.998669] Key type lgssc unregistered [14085.376892] LNet: 510146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14085.383739] LNetError: 510146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14085.396893] LNet: Removed LNI 192.168.203.139@tcp [14086.273780] Key type .llcrypt unregistered [14086.276417] Key type ._llcrypt unregistered [14116.044978] Key type ._llcrypt registered [14116.048203] Key type .llcrypt registered [14116.177076] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14118.038911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14118.053853] alg: No test for adler32 (adler32-zlib) [14119.219261] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [14119.544142] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [14121.495216] Key type lgssc registered [14124.003451] Lustre: Echo OBD driver; http://www.lustre.org/ [14133.490452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14133.511728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14134.970281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14140.440780] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14144.486259] LustreError: 511440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14144.500373] LustreError: 511440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14149.270975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14149.591453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14154.297552] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14161.191547] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14166.445378] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14172.355916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14172.653065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14178.193531] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14185.794466] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14197.251511] loop0: detected capacity change from 0 to 204800000 [14197.477354] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14197.486300] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14197.496540] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14203.280071] loop0: detected capacity change from 0 to 204800000 [14203.526271] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14203.545367] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14203.560200] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14208.776442] loop0: detected capacity change from 0 to 204800000 [14214.452063] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14214.615150] Lustre: fs15246-OST0000: new disk, initializing [14214.622393] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14214.634580] Lustre: 515029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14214.713943] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14221.233627] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14230.363792] loop1: detected capacity change from 0 to 204800000 [14234.987303] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14235.069821] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14235.141930] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14235.196562] Lustre: fs15246-MDT0000: new disk, initializing [14235.303261] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14235.333640] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14235.439279] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14236.514388] Lustre: 516153:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000269e0380 x1877070798959104/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790114344 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14238.812579] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14245.608833] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14245.617510] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14245.647992] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14260.719898] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14260.734097] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14264.668630] Lustre: server umount fs15246-MDT0000 complete [14265.839028] LustreError: 516571:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14270.951821] LustreError: 516571:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14274.689495] Lustre: server umount fs15246-OST0000 complete [14287.331344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14287.334834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14287.335946] LustreError: Skipped 1 previous similar message [14287.347048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14287.361473] Lustre: Skipped 1 previous similar message [14287.392023] Lustre: Skipped 1 previous similar message [14291.428358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14292.756433] Lustre: server umount lustre-OST0000 complete [14298.081356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14298.088284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14298.099944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14298.105658] Lustre: Skipped 1 previous similar message [14301.667639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14306.788237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14306.796320] Lustre: Skipped 3 previous similar messages [14311.391438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14311.578714] Lustre: server umount lustre-MDT0000 complete [14311.908279] LustreError: 511441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14311.931441] LustreError: 511441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14316.010269] LustreError: 511421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790114407 with bad export cookie 15408681499962483110 [14316.011681] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14316.022744] LustreError: 511421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14316.383533] Lustre: server umount lustre-MDT0001 complete [14328.174345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14331.954535] Key type lgssc unregistered [14332.296716] LNet: 519252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14332.303211] LNetError: 519252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14332.318571] LNet: Removed LNI 192.168.203.139@tcp [14333.385689] Key type .llcrypt unregistered [14333.387964] Key type ._llcrypt unregistered [14351.101994] Key type ._llcrypt registered [14351.106467] Key type .llcrypt registered [14351.206767] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14352.241900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14352.304282] alg: No test for adler32 (adler32-zlib) [14353.383450] Lustre: Lustre: Build Version: 2.17.58_63_g8caac7f [14353.683986] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [14355.329306] Key type lgssc registered [14356.399995] Lustre: Echo OBD driver; http://www.lustre.org/ [14391.332865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [14400.346957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14431.057741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14431.126520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14432.335434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14432.359743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14432.415422] Lustre: lustre-MDT0000: new disk, initializing [14432.514177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14432.550693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14437.619067] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14449.228228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14449.377768] Lustre: 523744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14449.488579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14449.493984] Lustre: Skipped 1 previous similar message [14449.629642] Lustre: lustre-MDT0001: new disk, initializing [14449.776186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14449.842734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14449.861272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14454.423610] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14465.375141] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14472.377705] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14480.024144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14480.183136] Lustre: lustre-OST0000: new disk, initializing [14480.190970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14480.195996] Lustre: 525712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14480.284061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14487.330765] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [14487.627359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14487.638908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14487.833520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14497.553786] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14502.800676] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14503.103064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14508.480202] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14508.775366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14513.648168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14513.683629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14513.685261] Lustre: Skipped 1 previous similar message [14513.692423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14513.725073] Lustre: Skipped 1 previous similar message [14517.024272] Lustre: server umount lustre-OST0000 complete [14523.873429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14523.884132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14523.892828] Lustre: Skipped 1 previous similar message [14528.180340] Lustre: server umount lustre-MDT0000 complete [14532.587258] LustreError: 523735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790114624 with bad export cookie 4060456644807333771 [14532.588970] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14532.603591] LustreError: 523735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14533.138706] Lustre: server umount lustre-MDT0001 complete [14559.308119] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [14569.711777] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [14601.835543] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14355 sec ========= 18:04:52 (1790114692) [14603.440927] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:04:54 (1790114694) === [14607.255525] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:04:57 (1790114697) === [14648.186701] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [14651.263241] Key type lgssc unregistered [14651.610786] LNet: 532354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14651.621647] LNetError: 532354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14651.643234] LNet: Removed LNI 192.168.203.139@tcp [14652.589720] Key type .llcrypt unregistered [14652.592833] Key type ._llcrypt unregistered