[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487909453 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.005003] kvm-guest: setup PV IPIs [ 0.007373] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.020536] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025221] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027098] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] *** VALIDATE tmpfs *** [ 0.031361] *** VALIDATE proc *** [ 0.032221] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036033] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042364] debug: unmapping init [mem 0xffffffff91c59000-0xffffffff91c60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053282] rcu: Hierarchical SRCU implementation. [ 0.055403] smp: Bringing up secondary CPUs ... [ 0.056589] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.060061] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134330] node 0 deferred pages initialised in 69ms [ 0.138098] devtmpfs: initialized [ 0.139221] x86/mm: Memory block size: 128MB [ 0.141690] gcov: version magic: 0x41383552 [ 0.144311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146267] pinctrl core: initialized pinctrl subsystem [ 0.147177] [ 0.147765] ************************************************************* [ 0.148012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149016] ** ** [ 0.150013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151012] ** ** [ 0.152012] ** This means that this kernel is built to expose internal ** [ 0.153011] ** IOMMU data structures, which may compromise security on ** [ 0.154012] ** your system. ** [ 0.155016] ** ** [ 0.156013] ** If you see this message and you are not debugging the ** [ 0.157015] ** kernel, report this immediately to your vendor! ** [ 0.158011] ** ** [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ************************************************************* [ 0.161696] NET: Registered protocol family 16 [ 0.162437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165450] cpuidle: using governor menu [ 0.166786] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167514] PCI: Using configuration type 1 for base access [ 0.168135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.177336] cryptd: max_cpu_qlen set to 1000 [ 0.180064] ACPI: Added _OSI(Module Device) [ 0.182015] ACPI: Added _OSI(Processor Device) [ 0.183014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.185015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.191480] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198631] ACPI: Interpreter enabled [ 0.199054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200011] ACPI: Using IOAPIC for interrupt routing [ 0.201088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202363] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211592] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213084] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.219073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.225389] acpiphp: Slot [2] registered [ 0.226211] acpiphp: Slot [5] registered [ 0.228118] acpiphp: Slot [6] registered [ 0.229122] acpiphp: Slot [7] registered [ 0.230110] acpiphp: Slot [8] registered [ 0.231152] acpiphp: Slot [9] registered [ 0.233142] acpiphp: Slot [10] registered [ 0.234115] acpiphp: Slot [3] registered [ 0.235118] acpiphp: Slot [4] registered [ 0.237089] acpiphp: Slot [11] registered [ 0.238090] acpiphp: Slot [12] registered [ 0.239087] acpiphp: Slot [13] registered [ 0.240084] acpiphp: Slot [14] registered [ 0.242101] acpiphp: Slot [15] registered [ 0.243099] acpiphp: Slot [16] registered [ 0.244188] acpiphp: Slot [17] registered [ 0.246100] acpiphp: Slot [18] registered [ 0.247157] acpiphp: Slot [19] registered [ 0.249195] acpiphp: Slot [20] registered [ 0.251112] acpiphp: Slot [21] registered [ 0.253384] acpiphp: Slot [22] registered [ 0.255101] acpiphp: Slot [23] registered [ 0.256098] acpiphp: Slot [24] registered [ 0.258131] acpiphp: Slot [25] registered [ 0.259290] acpiphp: Slot [26] registered [ 0.261110] acpiphp: Slot [27] registered [ 0.263113] acpiphp: Slot [28] registered [ 0.265117] acpiphp: Slot [29] registered [ 0.266127] acpiphp: Slot [30] registered [ 0.268114] acpiphp: Slot [31] registered [ 0.270079] PCI host bridge to bus 0000:00 [ 0.271018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.280054] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.285189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.291412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.301015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.306053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311069] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.315022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.318491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.320816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328754] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344973] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356086] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.403019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415465] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.464774] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596272] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.625017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.639017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653620] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.671017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.681022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.726670] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.737017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.743018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.766018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.775000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.776390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.779315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.781359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.784259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.788205] iommu: Default domain type: Passthrough [ 0.789411] SCSI subsystem initialized [ 0.791141] ACPI: bus type USB registered [ 0.793139] usbcore: registered new interface driver usbfs [ 0.795063] usbcore: registered new interface driver hub [ 0.796051] usbcore: registered new device driver usb [ 0.797117] pps_core: LinuxPPS API ver. 1 registered [ 0.799007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.801046] PTP clock support registered [ 0.803101] EDAC MC: Ver: 3.0.0 [ 0.805445] PCI: Using ACPI for IRQ routing [ 0.807474] NetLabel: Initializing [ 0.809012] NetLabel: domain hash size = 128 [ 0.809968] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.811059] NetLabel: unlabeled traffic allowed by default [ 0.813127] vgaarb: loaded [ 0.814247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.816017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.823983] clocksource: Switched to clocksource kvm-clock [ 0.929411] VFS: Disk quotas dquot_6.6.0 [ 0.931083] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934297] *** VALIDATE ramfs *** [ 0.935677] *** VALIDATE hugetlbfs *** [ 0.937277] pnp: PnP ACPI init [ 0.939567] pnp: PnP ACPI: found 6 devices [ 0.957418] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.960461] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.962429] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.964641] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.967105] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.969600] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.972527] NET: Registered protocol family 2 [ 0.974868] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.983361] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.988258] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.992332] TCP: Hash tables configured (established 65536 bind 65536) [ 0.994822] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.997505] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000514] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002899] NET: Registered protocol family 1 [ 1.006269] RPC: Registered named UNIX socket transport module. [ 1.008171] RPC: Registered udp transport module. [ 1.009799] RPC: Registered tcp transport module. [ 1.011524] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.013243] NET: Registered protocol family 44 [ 1.014624] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.017652] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.019266] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.021259] PCI: CLS 0 bytes, default 64 [ 1.023073] Unpacking initramfs... [ 2.400332] debug: unmapping init [mem 0xffff8fc2bcc54000-0xffff8fc2bffbffff] [ 2.404450] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.406594] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.409406] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.939286] Initialise system trusted keyrings [ 2.940972] Key type blacklist registered [ 2.942749] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.954222] zbud: loaded [ 2.958291] *** VALIDATE nfs *** [ 2.959442] *** VALIDATE nfs4 *** [ 2.961084] pstore: using deflate compression [ 2.964054] Platform Keyring initialized [ 3.046214] NET: Registered protocol family 38 [ 3.047274] Key type asymmetric registered [ 3.048291] Asymmetric key parser 'x509' registered [ 3.049422] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.051320] io scheduler mq-deadline registered [ 3.052387] io scheduler kyber registered [ 3.053450] io scheduler bfq registered [ 3.054518] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.056432] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.058302] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.060161] ACPI: Power Button [PWRF] [ 3.063503] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.067790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.079306] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.085397] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.099444] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.125594] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.156745] Non-volatile memory driver v1.3 [ 3.158291] Linux agpgart interface v0.103 [ 3.188054] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.189920] vda: detected capacity change from 0 to 69001216 [ 3.199814] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.201767] vdb: detected capacity change from 0 to 1073741824 [ 3.212070] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214220] vdc: detected capacity change from 0 to 2621440000 [ 3.226776] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228582] vdd: detected capacity change from 0 to 2621440000 [ 3.241107] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.243390] vde: detected capacity change from 0 to 4294967296 [ 3.255488] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.257326] vdf: detected capacity change from 0 to 4294967296 [ 3.261584] libphy: Fixed MDIO Bus: probed [ 3.278371] usbcore: registered new interface driver usbserial_generic [ 3.280430] usbserial: USB Serial support registered for generic [ 3.281922] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.284618] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.285778] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.287847] mousedev: PS/2 mouse device common for all mice [ 3.291158] rtc_cmos 00:05: RTC can wake from S4 [ 3.293234] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.297145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.297152] rtc_cmos 00:05: registered as rtc0 [ 3.300600] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.301054] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.305318] intel_pstate: CPU model not supported [ 3.308807] hid: raw HID events driver (C) Jiri Kosina [ 3.311291] usbcore: registered new interface driver usbhid [ 3.313334] usbhid: USB HID core driver [ 3.314780] drop_monitor: Initializing network drop monitor service [ 3.316928] Initializing XFRM netlink socket [ 3.318884] NET: Registered protocol family 10 [ 3.321834] Segment Routing with IPv6 [ 3.323165] NET: Registered protocol family 17 [ 3.325473] mpls_gso: MPLS GSO support [ 3.332377] RAS: Correctable Errors collector initialized. [ 3.334266] AVX version of gcm_enc/dec engaged. [ 3.336033] AES CTR mode by8 optimization enabled [ 3.414475] sched_clock: Marking stable (3414450955, 0)->(4329944489, -915493534) [ 3.419510] registered taskstats version 1 [ 3.421340] Loading compiled-in X.509 certificates [ 3.423589] zswap: loaded using pool lzo/zbud [ 3.446815] Key type big_key registered [ 3.458656] Key type encrypted registered [ 3.460062] ima: No TPM chip found, activating TPM-bypass! [ 3.461928] ima: Allocated hash algorithm: sha1 [ 3.463435] ima: No architecture policies found [ 3.464832] evm: Initialising EVM extended attributes: [ 3.466538] evm: security.selinux [ 3.467644] evm: security.ima [ 3.468784] evm: security.capability [ 3.469858] evm: HMAC attrs: 0x1 [ 3.472086] rtc_cmos 00:05: setting system clock to 2026-05-25 08:42:13 UTC (1779698533) [ 3.477922] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.481199] debug: unmapping init [mem 0xffffffff91982000-0xffffffff91c58fff] [ 3.490080] Write protecting the kernel read-only data: 28672k [ 3.492943] debug: unmapping init [mem 0xffffffff90003000-0xffffffff901fffff] [ 3.495404] debug: unmapping init [mem 0xffffffff90914000-0xffffffff909fffff] [ 3.522705] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.529736] systemd[1]: Detected virtualization kvm. [ 3.531091] systemd[1]: Detected architecture x86-64. [ 3.532632] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.558335] systemd[1]: No hostname configured. [ 3.559879] systemd[1]: Set hostname to . [ 3.561849] random: systemd: uninitialized urandom read (16 bytes read) [ 3.564130] systemd[1]: Initializing machine ID from random generator. [ 3.677793] random: systemd: uninitialized urandom read (16 bytes read) [ 3.680738] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.686535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.689205] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.693315] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.258293] device-mapper: uevent: version 1.0.3 [ 4.260600] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.955577] random: fast init done [ 4.960941] virtio_net virtio0 ens2: renamed from eth0 [ 5.115275] scsi host0: ata_piix [ 5.157218] scsi host1: ata_piix [ 5.159270] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.161220] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.678678] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.974584] random: crng init done [ 9.976032] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.254614] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.288159] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.523314] SELinux: Disabled at runtime. [ 11.588162] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.594827] systemd[1]: Detected virtualization kvm. [ 11.596672] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.035649] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.039169] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.044168] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.048299] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.051786] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.060884] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.069422] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ 12.152319] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.504292] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.818725] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.820242] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.933041] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.956306] EDAC sbridge: Ver: 1.1.2 [ 14.600741] Key type dns_resolver registered [ 14.940146] NFS: Registering the id_resolver key type [ 14.942180] Key type id_resolver registered [ 14.943803] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg224-server login: [ 63.610532] hrtimer: interrupt took 8040626 ns [ 86.702840] libcfs: loading out-of-tree module taints kernel. [ 86.756280] Key type ._llcrypt registered [ 86.758355] Key type .llcrypt registered [ 86.912650] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 107.351388] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 108.913402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.926119] alg: No test for adler32 (adler32-zlib) [ 110.578596] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 111.383418] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 113.143197] Key type lgssc registered [ 115.097759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.556742] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.535807] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 196.141736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.176740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.460194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.499734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.600892] Lustre: lustre-MDT0000: new disk, initializing [ 197.739773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.767753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.323842] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.865300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.095807] Lustre: 6500:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.142782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.149733] Lustre: Skipped 1 previous similar message [ 219.250459] Lustre: lustre-MDT0001: new disk, initializing [ 219.365948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.412118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.425067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.406565] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.978293] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.891484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.539232] Lustre: lustre-OST0000: new disk, initializing [ 245.549392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.559548] Lustre: 8405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.738624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.966879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.981389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.062877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.957547] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.580462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.730797] Lustre: lustre-OST0001: new disk, initializing [ 268.742074] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.748304] Lustre: 9459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.846480] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.739399] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.516208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.525404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.608961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.850995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.360926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 305.788493] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.891031] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing yml_node [ 317.058770] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 320.310196] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 323.369662] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 325.431882] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 04:47:33 EDT 2026 [ 347.925541] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 349.851719] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 367.584266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.584971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.589132] Lustre: Skipped 1 previous similar message [ 367.595342] Lustre: Skipped 2 previous similar messages [ 372.682645] Lustre: server umount lustre-MDT0000 complete [ 377.055637] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698907 with bad export cookie 13775413070638861651 [ 377.057464] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.065381] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 377.823580] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.824979] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.825700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.834466] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 377.841664] Lustre: Skipped 3 previous similar messages [ 382.944636] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.944654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.957367] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 382.985451] Lustre: Skipped 2 previous similar messages [ 383.723583] Lustre: server umount lustre-MDT0001 complete [ 388.072971] Lustre: server umount lustre-OST0000 complete [ 391.976208] Lustre: server umount lustre-OST0001 complete [ 399.364354] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 409.949908] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 449.661216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.949658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.987144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.094429] Lustre: lustre-MDT0000: new disk, initializing [ 450.216832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.239574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.874643] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 466.048894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.226380] Lustre: 14980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.299963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.303651] Lustre: Skipped 1 previous similar message [ 466.405474] Lustre: lustre-MDT0001: new disk, initializing [ 466.530594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.584610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.599940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.354546] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 478.980726] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 483.627631] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 490.892362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.095181] Lustre: lustre-OST0000: new disk, initializing [ 491.098676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.104293] Lustre: 16721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.191248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.580929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.590089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.780702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 498.069498] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 506.477159] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 511.429073] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.620416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.804544] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 516.990598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.750351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.753818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.779255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.793669] Lustre: Skipped 1 previous similar message [ 528.871035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.875808] Lustre: Skipped 2 previous similar messages [ 533.984682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 534.279267] Lustre: server umount lustre-OST0000 complete [ 539.105089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.109202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.115451] Lustre: Skipped 1 previous similar message [ 539.126888] Lustre: Skipped 3 previous similar messages [ 545.251071] Lustre: server umount lustre-MDT0000 complete [ 549.346375] LustreError: 14994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.377370] LustreError: 14994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 549.589660] LustreError: 14974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699079 with bad export cookie 13775413070638863184 [ 549.604170] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.157522] Lustre: server umount lustre-MDT0001 complete [ 555.106964] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:51:23 (1779699083) [ 557.937766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 560.256880] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:51:28 (1779699088) [ 568.618296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.215579] LustreError: 18985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.275511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.208276] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 574.441930] LustreError: 18986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.278133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.359310] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 595.345300] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.139393] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 608.840127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.992815] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 623.266568] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 638.944684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 638.953053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.961580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 638.961590] Lustre: Skipped 2 previous similar messages [ 638.993843] Lustre: Skipped 1 previous similar message [ 650.212885] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.495983] Lustre: server umount lustre-OST0000 complete [ 655.329935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.341253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.494914] Lustre: server umount lustre-MDT0000 complete [ 664.548885] LustreError: 18986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.574946] LustreError: 18986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 667.134843] LustreError: 18967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699197 with bad export cookie 13775413070638864066 [ 667.165721] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.170663] LustreError: 18967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 667.686961] Lustre: server umount lustre-MDT0001 complete [ 680.119350] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 683.563850] Key type lgssc unregistered [ 683.942762] LNet: 22184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.953361] LNetError: 22184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.979941] LNet: Removed LNI 192.168.202.124@tcp [ 685.032212] Key type .llcrypt unregistered [ 685.033974] Key type ._llcrypt unregistered [ 713.192919] Key type ._llcrypt registered [ 713.197362] Key type .llcrypt registered [ 713.349939] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 716.053947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.178500] alg: No test for adler32 (adler32-zlib) [ 717.494516] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 717.958711] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 719.687210] Key type lgssc registered [ 721.745462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 730.582449] loop: module loaded [ 730.695912] loop0: detected capacity change from 0 to 204800000 [ 730.928551] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.936191] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 731.131899] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.632823] loop0: detected capacity change from 0 to 204800000 [ 738.980613] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.008910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.053262] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.101038] loop0: detected capacity change from 0 to 204800000 [ 750.528132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.555561] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.845563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 751.942591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.076723] Lustre: lustre-MDT0000: new disk, initializing [ 752.180979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.213471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 758.274877] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 770.111765] loop1: detected capacity change from 0 to 204800000 [ 775.750767] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 775.921574] Lustre: lustre-OST07c6: new disk, initializing [ 775.925364] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 775.930065] Lustre: Skipped 1 previous similar message [ 775.934181] Lustre: 25337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 776.068317] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 784.248699] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 785.074190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 785.103935] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 785.159143] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 790.090677] Lustre: 24340:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 790.113062] Lustre: 24340:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 790.121545] Lustre: 24340:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 790.134291] Lustre: 24340:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 790.146174] Lustre: 24340:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 790.173836] Lustre: 24340:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 807.506216] Lustre: 24340:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 807.524095] Lustre: 24340:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.538834] Lustre: 24340:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.548050] Lustre: 24340:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 807.560533] Lustre: 24340:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 807.568770] Lustre: 24340:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.085197] Lustre: 24342:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 813.098302] Lustre: 24342:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.112060] Lustre: 24342:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.119439] Lustre: 24342:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.124730] Lustre: 24342:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.133822] Lustre: 24342:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.155858] Lustre: 24342:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 813.167597] Lustre: 24342:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.174066] Lustre: 24342:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 813.182740] Lustre: 24342:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.194117] Lustre: 24342:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.202519] Lustre: 24342:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.477530] Lustre: 24341:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 816.488664] Lustre: 24341:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 816.499272] Lustre: 24341:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 816.506212] Lustre: 24341:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.513675] Lustre: 24341:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.526758] Lustre: 24341:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.543247] Lustre: 24341:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 816.556587] Lustre: 24341:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.575074] Lustre: 24341:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 816.589372] Lustre: 24341:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.594272] Lustre: 24341:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.602013] Lustre: 24341:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.994023] Lustre: 24342:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 822.011408] Lustre: 24342:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 822.020321] Lustre: 24342:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 822.029462] Lustre: 24342:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.048376] Lustre: 24342:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 822.061227] Lustre: 24342:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.074834] Lustre: 24342:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 822.084198] Lustre: 24342:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.089173] Lustre: 24342:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 822.096308] Lustre: 24342:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.102230] Lustre: 24342:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 822.107244] Lustre: 24342:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.535512] Lustre: 24340:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 830.545680] Lustre: 24340:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 830.551720] Lustre: 24340:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 830.558473] Lustre: 24340:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.569315] Lustre: 24340:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.577227] Lustre: 24340:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.583820] Lustre: 24340:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 830.589886] Lustre: 24340:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.595217] Lustre: 24340:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 830.600236] Lustre: 24340:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.606864] Lustre: 24340:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.619735] Lustre: 24340:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 882.160222] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.169051] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.586817] Lustre: server umount lustre-OST07c6 complete [ 899.323904] Lustre: server umount lustre-MDT0000 complete [ 920.193348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 923.524431] Key type lgssc unregistered [ 923.859666] LNet: 27863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.884341] LNetError: 27863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.899954] LNet: Removed LNI 192.168.202.124@tcp [ 924.872730] Key type .llcrypt unregistered [ 924.876128] Key type ._llcrypt unregistered [ 944.143906] Key type ._llcrypt registered [ 944.146477] Key type .llcrypt registered [ 944.280637] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 945.113377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.148506] alg: No test for adler32 (adler32-zlib) [ 946.248503] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 946.480946] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 948.159456] Key type lgssc registered [ 949.282592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 958.006748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 958.028496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.834087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.777500] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 969.187825] LustreError: 29129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.217666] LustreError: 29129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 974.309228] LustreError: 29130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.439555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.801401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.168932] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 991.863505] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 996.672937] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1004.416936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.527955] LustreError: 30726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.947396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.168633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1011.910472] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1018.587838] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1035.744473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.756690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.768465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.768842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.776646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.777805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.970390] Lustre: server umount lustre-OST0000 complete [ 1047.018544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.026250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.034104] Lustre: Skipped 1 previous similar message [ 1047.052750] Lustre: Skipped 2 previous similar messages [ 1052.130031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.139829] Lustre: Skipped 1 previous similar message [ 1056.226269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.233414] Lustre: Skipped 1 previous similar message [ 1058.271259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.635479] Lustre: server umount lustre-MDT0000 complete [ 1061.351531] LustreError: 29129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.376228] LustreError: 29129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1062.769561] LustreError: 29110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699592 with bad export cookie 18104151547670699735 [ 1062.809462] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.257819] Lustre: server umount lustre-MDT0001 complete [ 1073.886795] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1076.659654] Key type lgssc unregistered [ 1076.930126] LNet: 32510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.943233] LNetError: 32510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.959580] LNet: Removed LNI 192.168.202.124@tcp [ 1077.798227] Key type .llcrypt unregistered [ 1077.800049] Key type ._llcrypt unregistered [ 1093.491756] Key type ._llcrypt registered [ 1093.494816] Key type .llcrypt registered [ 1093.610857] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1095.019364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.049512] alg: No test for adler32 (adler32-zlib) [ 1096.119963] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1096.347349] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1098.039595] Key type lgssc registered [ 1099.300083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.411891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.433903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.985463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.927244] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1118.183889] LustreError: 33743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.199390] LustreError: 33743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1121.619038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.077185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.426678] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1131.765170] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1135.308762] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.292432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.711567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.610807] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1153.020180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1154.500448] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1167.328709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.334814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.349620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.354905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.361872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.373431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.783693] Lustre: server umount lustre-OST0000 complete [ 1178.593333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1178.608053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.611802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.611811] Lustre: Skipped 1 previous similar message [ 1178.644879] Lustre: Skipped 1 previous similar message [ 1181.067397] Lustre: server umount lustre-MDT0000 complete [ 1183.725198] LustreError: 35369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.741994] LustreError: 35369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1185.740298] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699715 with bad export cookie 8276760777636208245 [ 1185.755792] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.756710] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1186.043636] Lustre: server umount lustre-MDT0001 complete [ 1196.264091] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1199.591291] Key type lgssc unregistered [ 1199.932760] LNet: 36930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.940762] LNetError: 36930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.964891] LNet: Removed LNI 192.168.202.124@tcp [ 1201.076434] Key type .llcrypt unregistered [ 1201.080169] Key type ._llcrypt unregistered [ 1231.277239] Key type ._llcrypt registered [ 1231.279127] Key type .llcrypt registered [ 1231.398561] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1232.897701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.966737] alg: No test for adler32 (adler32-zlib) [ 1234.173184] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1234.430830] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1236.119212] Key type lgssc registered [ 1237.269971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1245.565538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1245.588449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.029353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.820989] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1257.447028] LustreError: 38332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.464777] LustreError: 38332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1259.140612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.452198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.799565] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1270.131906] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1274.744572] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.043628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.609955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.980544] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1292.795880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1297.501521] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1309.172714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.177485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1309.192136] Lustre: Skipped 1 previous similar message [ 1309.200717] Lustre: Skipped 2 previous similar messages [ 1312.812302] Lustre: server umount lustre-MDT0000 complete [ 1314.289766] LustreError: 38328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.316403] LustreError: 38328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1317.222078] LustreError: 38312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699847 with bad export cookie 3335107449148406881 [ 1317.227992] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.238894] LustreError: 38312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1319.394350] LustreError: 39688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.402196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.417803] LustreError: 39688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1319.448686] Lustre: Skipped 1 previous similar message [ 1319.465585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1319.473958] Lustre: Skipped 1 previous similar message [ 1322.400111] LustreError: 39688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.400340] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1322.424188] LustreError: 39688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1322.432535] Lustre: Skipped 1 previous similar message [ 1323.838906] Lustre: server umount lustre-MDT0001 complete [ 1339.840573] Lustre: server umount lustre-OST0000 complete [ 1356.956827] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1360.122249] Key type lgssc unregistered [ 1360.582497] LNet: 41839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.589288] LNetError: 41839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.623827] LNet: Removed LNI 192.168.202.124@tcp [ 1361.491161] Key type .llcrypt unregistered [ 1361.492932] Key type ._llcrypt unregistered [ 1391.454510] Key type ._llcrypt registered [ 1391.455893] Key type .llcrypt registered [ 1391.625475] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1393.163190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.324065] alg: No test for adler32 (adler32-zlib) [ 1394.587245] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1394.922716] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1396.631238] Key type lgssc registered [ 1398.138193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.130935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.163888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.830640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.525937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1419.234415] LustreError: 43242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.257346] LustreError: 43242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1423.396477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.006292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.979193] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1435.499389] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1439.874906] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1447.773258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.200889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.474029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1455.562635] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1461.168814] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1468.981576] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1469.931713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.943246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.953630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.956518] LustreError: Skipped 1 previous similar message [ 1470.002602] Lustre: Skipped 1 previous similar message [ 1474.080829] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1474.092554] Lustre: Skipped 1 previous similar message [ 1474.840372] Lustre: server umount lustre-OST0000 complete [ 1495.521137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.531883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.538487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.543272] Lustre: Skipped 1 previous similar message [ 1499.005795] Lustre: server umount lustre-MDT0000 complete [ 1500.648445] LustreError: 43241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.663143] LustreError: 43241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1503.034951] LustreError: 43222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700033 with bad export cookie 12629941087566639058 [ 1503.056059] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.568569] Lustre: server umount lustre-MDT0001 complete [ 1525.916570] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1529.059322] Key type lgssc unregistered [ 1529.382871] LNet: 46590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.392199] LNetError: 46590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.424957] LNet: Removed LNI 192.168.202.124@tcp [ 1530.514857] Key type .llcrypt unregistered [ 1530.516513] Key type ._llcrypt unregistered [ 1557.735444] Key type ._llcrypt registered [ 1557.741406] Key type .llcrypt registered [ 1557.879327] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1559.249565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.295387] alg: No test for adler32 (adler32-zlib) [ 1560.325959] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1560.638057] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1562.448249] Key type lgssc registered [ 1563.646677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1571.617994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1571.650096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.189643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.719147] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1583.587743] LustreError: 47992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.609067] LustreError: 47992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1584.783216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.109030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.166659] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1593.784313] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1597.033580] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1603.737170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.052199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.251963] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1615.895920] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1618.352204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1625.715307] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1628.909262] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1641.814748] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-25 5h09m29s [ 1643.583962] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1646.561498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1646.567671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.583235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.653538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.661279] Lustre: Skipped 4 previous similar messages [ 1649.175324] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 1649.188108] Lustre: Skipped 1 previous similar message [ 1652.027588] Lustre: server umount lustre-MDT0000 complete [ 1653.744084] LustreError: 47993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.768131] LustreError: 47993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1655.236252] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1659.895514] LustreError: 49012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.908855] LustreError: 49012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1662.864857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.031711] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.345334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.416029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1664.368387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1668.242768] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1671.650459] Lustre: DEBUG MARKER: Wait for df (36109) ... [ 1673.370791] Lustre: DEBUG MARKER: done [ 1678.816325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1678.835535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.864477] Lustre: Skipped 2 previous similar messages [ 1678.874346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1679.845235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.769847] Lustre: server umount lustre-OST0000 complete [ 1690.083907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.104430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1690.111277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.111289] Lustre: Skipped 2 previous similar messages [ 1700.323467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1700.330603] Lustre: Skipped 6 previous similar messages [ 1702.367394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1702.739235] Lustre: server umount lustre-MDT0000 complete [ 1705.389742] LustreError: 47993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.416495] LustreError: 47993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1706.512803] LustreError: 50279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700236 with bad export cookie 1339183841056125972 [ 1706.518848] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.529778] LustreError: 50279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1707.172445] Lustre: server umount lustre-MDT0001 complete [ 1717.286615] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1719.703076] Key type lgssc unregistered [ 1719.972472] LNet: 52551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.987992] LNetError: 52551:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.081790] LNet: Removed LNI 192.168.202.124@tcp [ 1721.786411] Key type .llcrypt unregistered [ 1721.788212] Key type ._llcrypt unregistered [ 1759.343112] Key type ._llcrypt registered [ 1759.348241] Key type .llcrypt registered [ 1759.431957] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1760.960390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.007522] alg: No test for adler32 (adler32-zlib) [ 1762.075846] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1762.282743] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1763.967206] Key type lgssc registered [ 1765.052831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.581065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.608953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.901229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1773.919880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1774.052626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.735028] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1786.875444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.964947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1786.986426] Lustre: 54743:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1787.353820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.359934] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1796.482494] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.238889] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1806.837325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.990833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1807.190384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.182068] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1816.660740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1816.876703] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1822.799144] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-25 5h12m31s [ 1824.219645] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1830.943068] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1833.179604] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1833.260171] LustreError: 56732:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1853.499671] Lustre: lustre-MDT0000: Client 5981e0a8-33a1-4c31-9477-da74202f4227 (at 192.168.202.24@tcp) reconnecting [ 1873.263685] LustreError: 56732:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1875.795727] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1878.446761] Lustre: DEBUG MARKER: done [ 1888.232975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.237973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1888.240669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1888.262147] Lustre: Skipped 1 previous similar message [ 1891.024417] Lustre: server umount lustre-OST0000 complete [ 1894.880693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1894.890845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.910652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.914962] Lustre: Skipped 1 previous similar message [ 1895.971894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.984701] Lustre: Skipped 2 previous similar messages [ 1900.004177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.009956] Lustre: Skipped 1 previous similar message [ 1901.083283] Lustre: server umount lustre-MDT0000 complete [ 1904.886621] LustreError: 54734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700434 with bad export cookie 16782224315182354327 [ 1904.894651] LustreError: 54734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1904.897820] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.273129] Lustre: server umount lustre-MDT0001 complete [ 1915.747426] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1918.224859] Key type lgssc unregistered [ 1918.590562] LNet: 58716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.601126] LNetError: 58716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.638986] LNet: Removed LNI 192.168.202.124@tcp [ 1919.481449] Key type .llcrypt unregistered [ 1919.484813] Key type ._llcrypt unregistered [ 1959.850841] Key type ._llcrypt registered [ 1959.862170] Key type .llcrypt registered [ 1959.972653] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1960.864976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.883624] alg: No test for adler32 (adler32-zlib) [ 1961.863412] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1962.077485] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1963.807163] Key type lgssc registered [ 1965.076807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1968.476940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1968.501868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.836538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1969.904029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1970.211593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.130855] Lustre: Failing over lustre-MDT0000 [ 1971.775707] Lustre: server umount lustre-MDT0000 complete [ 1978.722707] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:15:07 (1779700507) [ 1986.128391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.407479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.955503] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2000.078034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.133789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2000.155271] Lustre: 61497:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2000.388618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2005.182695] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2011.397535] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2015.241957] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2022.381071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.630115] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2022.863766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.247706] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2031.345368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2037.380653] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2044.631207] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2052.065614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.097795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.111092] Lustre: Skipped 1 previous similar message [ 2057.187394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.201962] Lustre: Skipped 2 previous similar messages [ 2062.328158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.339420] Lustre: Skipped 2 previous similar messages [ 2066.403832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2066.667661] Lustre: server umount lustre-MDT0000 complete [ 2071.007613] LustreError: 61489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700601 with bad export cookie 1909243518188755109 [ 2071.033677] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.553950] Lustre: server umount lustre-MDT0001 complete [ 2075.719073] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2080.475419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2088.992166] Lustre: 60419:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700602/real 1779700602] req@00000000e6b24be4 x1866151223141504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2089.026592] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.046371] Lustre: Skipped 2 previous similar messages [ 2092.703800] Lustre: 60420:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700606/real 1779700606] req@00000000b015e398 x1866151223141632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2093.536153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.819153] LustreError: 63117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.853235] LustreError: 63117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2097.010667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.160523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2097.953252] Lustre: 60422:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700611/real 1779700611] req@000000002c2a7815 x1866151223142016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2098.015754] Lustre: 60422:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2098.024867] LustreError: 63118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.034354] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2099.048428] LustreError: 63117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.070107] LustreError: 63117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2102.122626] Lustre: 60419:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700616/real 1779700616] req@00000000a8d5a9a9 x1866151223142144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2103.699498] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2104.298530] LustreError: 64931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.327316] LustreError: 64931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2109.431866] LustreError: 64930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.469391] LustreError: 64930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2115.801550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.291323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.658530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2121.731200] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2126.628844] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2131.112065] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2141.669715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2141.694974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.716663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.766858] Lustre: server umount lustre-MDT0000 complete [ 2147.298627] LustreError: 64929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.317486] LustreError: 64929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2150.264698] LustreError: 61489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700680 with bad export cookie 1909243518188757384 [ 2150.271677] LustreError: 61489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.274660] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.806518] Lustre: server umount lustre-MDT0001 complete [ 2158.079125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2167.787912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.735176] Lustre: 60421:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700682/real 1779700682] req@00000000d43546e0 x1866151223169792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.786375] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.800897] Lustre: Skipped 2 previous similar messages [ 2176.274491] LustreError: 63117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.375755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.461335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2176.991130] Lustre: 60421:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779700690/real 1779700690] req@00000000fc0b81ee x1866151223170304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779700706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.015225] Lustre: 60421:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2180.786317] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2190.792644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.870289] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2196.475705] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2201.659601] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2205.789469] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2216.929454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.944429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.969855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.989490] Lustre: Skipped 3 previous similar messages [ 2219.776574] Lustre: server umount lustre-MDT0000 complete [ 2223.073943] LustreError: 68365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.084437] LustreError: 68365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2223.718835] LustreError: 61489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700753 with bad export cookie 1909243518188758763 [ 2223.720872] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.737465] LustreError: 61489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2224.196795] Lustre: server umount lustre-MDT0001 complete [ 2230.331567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2234.301561] Lustre: DEBUG MARKER: files compared the same [ 2247.231651] Lustre: server umount lustre-OST0000 complete [ 2265.104926] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2267.773700] Key type lgssc unregistered [ 2268.060656] LNet: 70421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.074339] LNetError: 70421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.095192] LNet: Removed LNI 192.168.202.124@tcp [ 2268.831646] Key type .llcrypt unregistered [ 2268.834259] Key type ._llcrypt unregistered [ 2291.138591] Key type ._llcrypt registered [ 2291.141976] Key type .llcrypt registered [ 2291.304793] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2293.161960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.273489] alg: No test for adler32 (adler32-zlib) [ 2294.340668] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2294.509645] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2296.175168] Key type lgssc registered [ 2297.238591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2305.204176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2305.228451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.616935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.991929] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug +malloc all [ 2316.770107] LustreError: 71822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.778270] LustreError: 71822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2319.955211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.433894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.728406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug +malloc all [ 2330.666785] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.855501] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2341.944084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.261864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.968733] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug +malloc all [ 2351.476221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2354.360685] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2365.928831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2365.939455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.952383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.948390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.948765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.949148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2369.751692] Lustre: server umount lustre-OST0000 complete [ 2376.673867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.682977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.694598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.699223] Lustre: Skipped 1 previous similar message [ 2379.838524] Lustre: server umount lustre-MDT0000 complete [ 2382.304824] LustreError: 74178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.321701] LustreError: 74178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2383.103517] LustreError: 71802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700913 with bad export cookie 16923058717099676570 [ 2383.108467] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.111112] LustreError: 71802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2383.478768] Lustre: server umount lustre-MDT0001 complete [ 2393.282871] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2395.872096] Key type lgssc unregistered [ 2396.261679] LNet: 74996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.267931] LNetError: 74996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.291476] LNet: Removed LNI 192.168.202.124@tcp [ 2398.525949] Key type .llcrypt unregistered [ 2398.529471] Key type ._llcrypt unregistered [ 2426.282899] Key type ._llcrypt registered [ 2426.283979] Key type .llcrypt registered [ 2426.361329] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2427.097553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.119048] alg: No test for adler32 (adler32-zlib) [ 2428.171588] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2428.393642] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2430.095247] Key type lgssc registered [ 2431.565687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2438.852934] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2438.876858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.431345] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2459.551244] LustreError: 76372:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 2486.246751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.814414] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2496.661508] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2503.694470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.799675] Lustre: *** cfs_fail_loc=706, val=0*** [ 2504.134459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.209833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2507.909194] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2509.284941] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.315398] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2514.402806] LustreError: 76398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.988353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.239861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.583807] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2523.430964] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2527.193231] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2529.761608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.768754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.788833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2530.788468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2530.799985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.823704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2535.785799] Lustre: server umount lustre-OST0000 complete [ 2541.024502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2541.027790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.050903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.059694] Lustre: Skipped 2 previous similar messages [ 2547.169168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.182215] Lustre: Skipped 3 previous similar messages [ 2552.292744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.304094] Lustre: Skipped 1 previous similar message [ 2554.336970] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2554.558616] Lustre: server umount lustre-MDT0000 complete [ 2556.385468] LustreError: 77349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.395537] LustreError: 77349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2558.050707] LustreError: 76380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701088 with bad export cookie 12286505505525600944 [ 2558.063389] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.370399] Lustre: server umount lustre-MDT0001 complete [ 2567.190396] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2570.031664] Key type lgssc unregistered [ 2570.315390] LNet: 79629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2570.324223] LNetError: 79629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2570.357657] LNet: Removed LNI 192.168.202.124@tcp [ 2571.216580] Key type .llcrypt unregistered [ 2571.223725] Key type ._llcrypt unregistered [ 2592.691473] Key type ._llcrypt registered [ 2592.696951] Key type .llcrypt registered [ 2592.777153] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2593.770493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.849557] alg: No test for adler32 (adler32-zlib) [ 2595.100265] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2595.378297] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2597.071264] Key type lgssc registered [ 2598.170567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2606.197529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2606.218252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.448414] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2618.021540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.512178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2622.355382] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2623.974766] LustreError: 81698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.987988] LustreError: 81698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2628.066789] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.083684] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2630.213572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2630.477839] LustreError: 81698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.504792] LustreError: 81698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2630.522649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.760913] LustreError: 82427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.785853] LustreError: 82427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2636.840514] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2640.869590] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.903514] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2646.855333] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.872135] Lustre: MGS: Client 776794bd-c9b2-4754-b4e1-2de6e509c8c4 (at 0@lo) reconnecting [ 2646.900242] LustreError: 81712:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2647.227307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.333022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2651.824659] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2668.005071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2668.007525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.010511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2668.022661] Lustre: Skipped 1 previous similar message [ 2672.656580] Lustre: server umount lustre-OST0000 complete [ 2678.242494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.242557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2678.244671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.244701] Lustre: Skipped 1 previous similar message [ 2678.276284] Lustre: Skipped 1 previous similar message [ 2682.590393] Lustre: server umount lustre-MDT0000 complete [ 2683.360601] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.388919] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2686.258619] Lustre: server umount lustre-MDT0001 complete [ 2690.367891] Lustre: server umount MGS complete [ 2700.981350] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:27:09 (1779701229) [ 2730.242328] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 2741.399708] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2774.676803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.815152] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2788.054101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.436467] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2788.498327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2788.634678] Lustre: lustre-MDT0001: new disk, initializing [ 2788.725617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.756296] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2788.767795] Lustre: 88787:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2788.780053] LustreError: 88787:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2792.649197] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2803.535259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.789855] Lustre: lustre-OST0000: new disk, initializing [ 2803.793652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2803.802406] Lustre: 89535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2803.887732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.636289] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2813.088326] Lustre: 89945:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000f3b898d1 x1866151886888704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779701359 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2817.739579] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2824.050809] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.059475] Lustre: MGS: Client d2b79ee3-48d8-45e2-974f-b193b207054c (at 0@lo) reconnecting [ 2824.103336] Lustre: 88074:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2824.116735] LustreError: 88773:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2824.159775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2824.340656] Lustre: lustre-MDT0000: new disk, initializing [ 2824.550178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.597191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2824.676704] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2825.632585] Lustre: 90500:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000a475650e x1866151886895104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1779701371 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2828.915159] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2833.377681] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2834.737343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2834.743824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2834.900619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2860.015095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.018118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.032056] Lustre: Skipped 1 previous similar message [ 2860.039066] Lustre: Skipped 1 previous similar message [ 2863.075596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2863.078639] Lustre: Skipped 2 previous similar messages [ 2864.679831] Lustre: server umount lustre-OST0000 complete [ 2870.240626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2870.251236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.268472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.624463] Lustre: server umount lustre-MDT0000 complete [ 2884.675643] Lustre: server umount lustre-MDT0001 complete [ 2888.513699] Lustre: server umount MGS complete [ 2899.671156] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:30:28 (1779701428) [ 2915.981383] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2918.825963] Key type lgssc unregistered [ 2919.075069] LNet: 92964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.081459] LNetError: 92964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.096841] LNet: Removed LNI 192.168.202.124@tcp [ 2919.887947] Key type .llcrypt unregistered [ 2919.890665] Key type ._llcrypt unregistered [ 2936.724768] Key type ._llcrypt registered [ 2936.726574] Key type .llcrypt registered [ 2936.812589] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2938.019741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2938.042176] alg: No test for adler32 (adler32-zlib) [ 2939.095785] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2939.287341] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2940.943211] Key type lgssc registered [ 2941.966071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2949.532877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2949.550894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2951.096525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.354730] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2961.393498] LustreError: 94171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.416443] LustreError: 94171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2962.927074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.251276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2966.814590] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2971.305832] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2974.917552] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.424486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.744743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.354437] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2993.239349] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2996.086997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3009.509086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.518051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.543342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.559203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.561246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3011.572680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.105909] Lustre: server umount lustre-OST0000 complete [ 3019.744556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3019.751721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.775670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.785311] Lustre: Skipped 1 previous similar message [ 3021.798385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.801731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.814369] Lustre: Skipped 1 previous similar message [ 3023.070716] Lustre: server umount lustre-MDT0000 complete [ 3026.912960] LustreError: 94171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.928592] LustreError: 94171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3027.071065] LustreError: 94149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701557 with bad export cookie 1059877959940311804 [ 3027.072633] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.077787] LustreError: 94149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3027.479460] Lustre: server umount lustre-MDT0001 complete [ 3039.423786] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3042.372138] Key type lgssc unregistered [ 3042.677600] LNet: 97367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3042.681632] LNetError: 97367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3042.697167] LNet: Removed LNI 192.168.202.124@tcp [ 3043.408724] Key type .llcrypt unregistered [ 3043.413277] Key type ._llcrypt unregistered [ 3060.024069] Key type ._llcrypt registered [ 3060.026261] Key type .llcrypt registered [ 3060.174417] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3061.508228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.537809] alg: No test for adler32 (adler32-zlib) [ 3062.643894] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3062.857470] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3064.546185] Key type lgssc registered [ 3065.647701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3071.850251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3071.854606] LustreError: 98436:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3071.859753] Lustre: Skipped 1 previous similar message [ 3071.859772] LustreError: 98435:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3071.879518] LustreError: 98436:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3071.886793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.887039] LustreError: 98436:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3071.887055] LustreError: 98436:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3073.624666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.881219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.161883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.141206] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3091.811680] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing lsmod [ 3095.122468] LustreError: 99607:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3095.128813] LustreError: 99606:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3095.140060] LustreError: 99607:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3095.154415] LustreError: 99607:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3095.165067] LustreError: 99607:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3095.196348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.510799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.644610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.646575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3100.662982] Lustre: Skipped 2 previous similar messages [ 3100.683320] Lustre: Skipped 2 previous similar messages [ 3104.442120] Lustre: server umount lustre-MDT0000 complete [ 3105.762200] LustreError: 98474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.778995] LustreError: 98474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3107.825454] LustreError: 98468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.837862] LustreError: 98468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3108.149029] LustreError: 98449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701638 with bad export cookie 1838297962953732382 [ 3108.154830] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.160651] LustreError: 98449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3109.919630] LustreError: 98904:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000059a47c39 x1866152376553600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3110.261279] Lustre: server umount lustre-MDT0001 complete [ 3124.586470] Lustre: server umount lustre-OST0000 complete [ 3133.896580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.547550] LustreError: 100671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.587442] LustreError: 100671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3134.662542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.072725] LustreError: 100672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.107105] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3144.163283] LustreError: 100672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.502275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.777681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3155.431382] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3161.293425] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3165.588614] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3173.130568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.410110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.572850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3180.578574] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3188.415780] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3199.968789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.975404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.991469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3199.995246] Lustre: Skipped 1 previous similar message [ 3202.839737] Lustre: server umount lustre-OST0000 complete [ 3210.208318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.233092] Lustre: Skipped 1 previous similar message [ 3210.247933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.255164] Lustre: Skipped 1 previous similar message [ 3213.187230] Lustre: server umount lustre-MDT0000 complete [ 3215.342902] LustreError: 100668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.373969] LustreError: 100668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3217.123079] LustreError: 100651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701747 with bad export cookie 1838297962953733082 [ 3217.129312] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3217.139968] LustreError: 100651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3217.522736] Lustre: server umount lustre-MDT0001 complete [ 3227.535717] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3230.428414] Key type lgssc unregistered [ 3230.686555] LNet: 103847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.700054] LNetError: 103847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.719163] LNet: Removed LNI 192.168.202.124@tcp [ 3231.502641] Key type .llcrypt unregistered [ 3231.505104] Key type ._llcrypt unregistered [ 3254.118566] Key type ._llcrypt registered [ 3254.120340] Key type .llcrypt registered [ 3254.212937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3255.110518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3255.260245] alg: No test for adler32 (adler32-zlib) [ 3256.387608] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3256.666782] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3258.343239] Key type lgssc registered [ 3259.728705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3270.654857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.682472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.230213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.867625] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3282.410246] LustreError: 105246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.432387] LustreError: 105246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3283.435702] LustreError: 105247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.145565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.522587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.001074] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3297.170120] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.386755] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3309.369991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.887030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.054217] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3318.080895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3323.385712] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.063990] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3331.620796] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3337.712295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3337.721072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.733869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.720739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.723506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.728441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3338.773589] Lustre: Skipped 1 previous similar message [ 3342.520918] Lustre: server umount lustre-OST0000 complete [ 3347.937707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3347.953397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.000338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.499270] Lustre: server umount lustre-MDT0000 complete [ 3354.081942] LustreError: 106220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.118365] LustreError: 106220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3356.717321] LustreError: 107439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701886 with bad export cookie 16319505873932590605 [ 3356.719555] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.727960] LustreError: 107439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.287281] Lustre: server umount lustre-MDT0001 complete [ 3368.513971] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3371.333692] Key type lgssc unregistered [ 3371.567887] LNet: 108529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3371.572220] LNetError: 108529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3371.591380] LNet: Removed LNI 192.168.202.124@tcp [ 3372.335404] Key type .llcrypt unregistered [ 3372.339716] Key type ._llcrypt unregistered [ 3392.128922] Key type ._llcrypt registered [ 3392.141415] Key type .llcrypt registered [ 3392.267901] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3394.070833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.090583] alg: No test for adler32 (adler32-zlib) [ 3395.163289] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3395.507309] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3397.208306] Key type lgssc registered [ 3398.489097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.589587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3406.606979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.366535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.112743] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3418.595647] LustreError: 109735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.615395] LustreError: 109735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3422.139090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.680649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.190932] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3432.599633] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3437.096144] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.843282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3446.348658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.713243] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3453.616179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3458.644032] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3468.258860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.264156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.277126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.284154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.286697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.306988] Lustre: Skipped 1 previous similar message [ 3473.382450] Lustre: server umount lustre-OST0000 complete [ 3479.008330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.019651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.033719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.972086] Lustre: server umount lustre-MDT0000 complete [ 3484.643888] LustreError: 109732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.665585] LustreError: 109732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3486.774491] LustreError: 109716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702016 with bad export cookie 6380859394360283883 [ 3486.775368] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.785351] LustreError: 109716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3487.054317] Lustre: server umount lustre-MDT0001 complete [ 3501.322223] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3504.428092] Key type lgssc unregistered [ 3504.719720] LNet: 112913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3504.725306] LNetError: 112913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3504.755295] LNet: Removed LNI 192.168.202.124@tcp [ 3505.918961] Key type .llcrypt unregistered [ 3505.924383] Key type ._llcrypt unregistered [ 3532.158914] Key type ._llcrypt registered [ 3532.161295] Key type .llcrypt registered [ 3532.314103] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3533.397722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.584804] alg: No test for adler32 (adler32-zlib) [ 3534.715663] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3534.944464] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3536.599165] Key type lgssc registered [ 3537.675393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3546.591120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3546.663031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.290833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.246681] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3558.884525] LustreError: 114317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.898345] LustreError: 114317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3563.685522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.126808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3568.791192] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3575.144633] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3579.423146] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3586.222447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.525571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.423746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3593.730864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3598.799868] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3607.687618] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3611.531656] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3611.534428] Lustre: Skipped 2 previous similar messages [ 3615.850083] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3615.854539] Lustre: Skipped 2 previous similar messages [ 3617.971529] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3618.026900] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3623.742596] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3623.751169] Lustre: Skipped 2 previous similar messages [ 3627.910542] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.24@tcp 192.168.0.[2,10]@tcp [ 3632.411816] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.24/24@tcp in log lustre-MDT0000 [ 3632.424124] Lustre: Skipped 2 previous similar messages [ 3633.215200] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.24/24@tcp [ 3633.226405] Lustre: Skipped 2 previous similar messages [ 3640.800823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.808054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.817356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.830193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3641.837825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.839963] LustreError: Skipped 1 previous similar message [ 3641.855426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3644.900599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3644.907359] Lustre: Skipped 1 previous similar message [ 3646.459148] Lustre: server umount lustre-OST0000 complete [ 3650.016733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.034373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.043525] Lustre: Skipped 1 previous similar message [ 3655.138406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.148953] Lustre: Skipped 1 previous similar message [ 3664.351251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3664.643866] Lustre: server umount lustre-MDT0000 complete [ 3665.378805] LustreError: 114317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.413382] LustreError: 114317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3668.438410] LustreError: 117257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702198 with bad export cookie 15490992260296210744 [ 3668.440592] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.454095] LustreError: 117257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3669.138951] Lustre: server umount lustre-MDT0001 complete [ 3680.313859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3683.676594] Key type lgssc unregistered [ 3683.957096] LNet: 118653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3683.969936] LNetError: 118653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3684.011335] LNet: Removed LNI 192.168.202.124@tcp [ 3685.124585] Key type .llcrypt unregistered [ 3685.126328] Key type ._llcrypt unregistered [ 3719.793796] Key type ._llcrypt registered [ 3719.795426] Key type .llcrypt registered [ 3719.930793] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3721.442266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3721.550671] alg: No test for adler32 (adler32-zlib) [ 3722.556389] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3722.808873] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3724.487211] Key type lgssc registered [ 3726.615650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3736.384275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3736.409547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.163585] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3738.190727] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.24/24@tcp [ 3738.207910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.321245] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3748.323984] LustreError: 120200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.368928] LustreError: 120200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3753.446096] LustreError: 120201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.308710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.885890] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3754.936339] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.24/24@tcp [ 3755.035747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3761.087822] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3766.968188] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3770.537752] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3777.657090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.097520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.357206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3785.167886] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3791.879401] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3804.128246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.134718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.149113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.164086] Lustre: Skipped 1 previous similar message [ 3804.191439] Lustre: Skipped 1 previous similar message [ 3808.989666] Lustre: server umount lustre-OST0000 complete [ 3814.369227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.371906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3814.412016] Lustre: Skipped 1 previous similar message [ 3814.443866] Lustre: Skipped 1 previous similar message [ 3819.490707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.514704] LustreError: 120195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.662967] Lustre: server umount lustre-MDT0000 complete [ 3823.810149] LustreError: 120182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702353 with bad export cookie 3248381225614107537 [ 3823.821486] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3823.825040] LustreError: 120182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3824.328325] Lustre: server umount lustre-MDT0001 complete [ 3835.732405] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 3838.328197] Key type lgssc unregistered [ 3838.644545] LNet: 123431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3838.653533] LNetError: 123431:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3838.668468] LNet: Removed LNI 192.168.202.124@tcp [ 3839.656172] Key type .llcrypt unregistered [ 3839.658376] Key type ._llcrypt unregistered [ 3872.358919] Key type ._llcrypt registered [ 3872.362653] Key type .llcrypt registered [ 3872.520752] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 3889.484177] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3890.960357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3891.079073] alg: No test for adler32 (adler32-zlib) [ 3892.374557] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3892.621665] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 3894.383177] Key type lgssc registered [ 3895.621664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3932.490774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3932.552306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.864354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3933.908217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3934.061725] Lustre: lustre-MDT0000: new disk, initializing [ 3934.217420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.242835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3938.926278] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3950.806879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.966183] Lustre: 126800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3951.030745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3951.033883] Lustre: Skipped 1 previous similar message [ 3951.102314] Lustre: lustre-MDT0001: new disk, initializing [ 3951.212940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.266531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3951.278873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3956.015838] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3965.710175] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3970.496217] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3978.025308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3978.244296] Lustre: lustre-OST0000: new disk, initializing [ 3978.247660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3978.252944] Lustre: 128539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3978.334925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.516652] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3987.540912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3987.549304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3987.782080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3992.413671] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3996.943896] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3997.116787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4002.124707] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4002.418895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4008.416500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.422697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.426386] LustreError: Skipped 1 previous similar message [ 4008.429383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.441541] Lustre: Skipped 1 previous similar message [ 4008.489705] Lustre: Skipped 1 previous similar message [ 4011.368538] Lustre: server umount lustre-OST0000 complete [ 4017.632344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.640084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.655274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.657584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.658517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.676060] Lustre: Skipped 1 previous similar message [ 4022.751723] Lustre: server umount lustre-MDT0000 complete [ 4027.568268] LustreError: 129935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702557 with bad export cookie 7884516779141813590 [ 4027.593320] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.046020] Lustre: server umount lustre-MDT0001 complete [ 4037.631434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.992472] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.015037] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4038.057384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.400613] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4043.241037] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.259020] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4048.360368] LustreError: 130616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.390753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.942345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.435531] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4063.711346] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4068.484452] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4075.138223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.524557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.916805] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4091.518490] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4097.442353] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4097.624929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4102.787285] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4102.982792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4112.028296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.179327] Lustre: lustre-OST0001: new disk, initializing [ 4112.182856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4112.188610] Lustre: 133615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4112.283129] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.605323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4117.621418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4117.716792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4118.376936] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4126.252816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4126.460520] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4131.396122] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4131.666824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4136.318926] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ee1430d3000.ost_server_uuid 50 [ 4137.720271] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ee1430d3000.ost_server_uuid in FULL state after 0 sec [ 4148.705539] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4148.726595] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.745763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4148.748838] Lustre: Skipped 1 previous similar message [ 4150.869058] Lustre: server umount lustre-OST0001 complete [ 4153.829023] LustreError: 133889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.850224] LustreError: 133889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4155.364372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4155.376959] LustreError: Skipped 1 previous similar message [ 4155.380384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.404097] Lustre: Skipped 1 previous similar message [ 4155.410598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4158.945838] LustreError: 132212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.971632] LustreError: 132212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4161.416616] Lustre: server umount lustre-OST0000 complete [ 4169.185300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.195049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.212030] Lustre: Skipped 2 previous similar messages [ 4169.215093] Lustre: Skipped 2 previous similar messages [ 4173.600714] Lustre: server umount lustre-MDT0000 complete [ 4174.321250] LustreError: 131326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.354800] LustreError: 131326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4177.463682] LustreError: 130597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702707 with bad export cookie 7884516779141814465 [ 4177.466643] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.471657] LustreError: 130597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4178.165231] Lustre: server umount lustre-MDT0001 complete [ 4198.506463] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 4200.975749] Key type lgssc unregistered [ 4201.291036] LNet: 136453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.295805] LNetError: 136453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.308951] LNet: Removed LNI 192.168.202.124@tcp [ 4202.005108] Key type .llcrypt unregistered [ 4202.007187] Key type ._llcrypt unregistered [ 4241.181494] Key type ._llcrypt registered [ 4241.183737] Key type .llcrypt registered [ 4241.302298] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 4257.667393] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4259.450688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4259.490134] alg: No test for adler32 (adler32-zlib) [ 4260.568997] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 4260.853646] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 4262.503319] Key type lgssc registered [ 4263.550709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4307.908332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4307.933950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.252988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4309.296587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4309.417922] Lustre: lustre-MDT0000: new disk, initializing [ 4309.564942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.598389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4314.599044] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4327.162587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.392310] Lustre: 140575:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4327.511976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4327.526770] Lustre: Skipped 1 previous similar message [ 4327.640839] Lustre: lustre-MDT0001: new disk, initializing [ 4327.790187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4327.872295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4327.879501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4333.421049] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4342.042885] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4346.861558] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4354.724204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.100521] Lustre: lustre-OST0000: new disk, initializing [ 4355.106287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4355.112196] Lustre: 142317:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4355.205283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.046214] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4364.871238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4364.881373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4364.994925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4370.790341] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4374.898589] Lustre: Failing over lustre-OST0000 [ 4375.157986] Lustre: server umount lustre-OST0000 complete [ 4391.378593] Lustre: 138798:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702905/real 1779702905] req@00000000b27e6c22 x1866153632759808/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779702921 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4391.418616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.482702] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702905/real 1779702905] req@00000000f095ad5e x1866153632759552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779702921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4392.533555] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4393.951165] Lustre: 138796:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702907/real 1779702907] req@0000000030781f22 x1866153632760064/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1779702923 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4393.991288] Lustre: 138796:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4394.881226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.412732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.431869] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4396.383197] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702910/real 1779702910] req@000000007b0ecef5 x1866153632761088/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779702926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4396.431171] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4401.568162] Lustre: 138796:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702915/real 1779702915] req@0000000069656360 x1866153632762240/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779702931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4401.610579] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4403.278040] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4403.281740] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4403.443680] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4408.048904] Lustre: Failing over lustre-MDT0000 [ 4408.637891] Lustre: server umount lustre-MDT0000 complete [ 4409.825880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4409.842531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.870899] Lustre: Skipped 1 previous similar message [ 4411.879166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.925718] Lustre: Skipped 1 previous similar message [ 4413.484778] LustreError: 140581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.512241] LustreError: 140581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4416.993453] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.012463] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4418.601565] LustreError: 141681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.114620] LustreError: 140580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.169653] LustreError: 140580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4427.232630] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779702941/real 1779702941] req@0000000036b4a3ea x1866153632774656/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779702957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4427.240538] LustreError: 141681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.281166] Lustre: 138797:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4427.281266] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.366501] LustreError: 141681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4430.071374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.472300] LustreError: 138795:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c2b4ff42 x1866153632779392/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4437.502065] LustreError: 140585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.517937] LustreError: 140585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4437.913042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.966041] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4439.058181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4443.135488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4443.143030] Lustre: Skipped 1 previous similar message [ 4443.184817] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4443.252290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4444.026396] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4453.351132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.360903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.366484] Lustre: Skipped 1 previous similar message [ 4453.377367] Lustre: Skipped 1 previous similar message [ 4457.857758] Lustre: server umount lustre-OST0000 complete [ 4463.585236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4463.595640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.604193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.622552] Lustre: Skipped 1 previous similar message [ 4463.637466] Lustre: Skipped 1 previous similar message [ 4467.916930] Lustre: server umount lustre-MDT0000 complete [ 4468.713981] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.746317] LustreError: 140586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4472.079321] LustreError: 140566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703002 with bad export cookie 12964498584921952960 [ 4472.103500] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.583892] Lustre: server umount lustre-MDT0001 complete [ 4483.668408] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 4486.668091] Key type lgssc unregistered [ 4486.954420] LNet: 145814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4486.970704] LNetError: 145814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4486.993040] LNet: Removed LNI 192.168.202.124@tcp [ 4488.023344] Key type .llcrypt unregistered [ 4488.027768] Key type ._llcrypt unregistered [ 4527.981685] Key type ._llcrypt registered [ 4527.984784] Key type .llcrypt registered [ 4528.098490] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4529.423802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4529.533609] alg: No test for adler32 (adler32-zlib) [ 4530.687789] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 4530.969037] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 4532.687180] Key type lgssc registered [ 4534.699306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4546.626481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4546.649549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.309313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4553.186096] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4558.825216] LustreError: 147318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.845870] LustreError: 147318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4561.180332] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.944461] LustreError: 147317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.373850] LustreError: 147346:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4565.392526] LustreError: 147346:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4565.407309] LustreError: 147346:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4565.981362] Lustre: server umount lustre-MDT0000 complete [ 4588.431594] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 4599.976429] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4641.427457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.707294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4641.749359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4641.847657] Lustre: lustre-MDT0000: new disk, initializing [ 4641.932893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.971597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4647.344394] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4661.130539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.380974] Lustre: 150503:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4661.461739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4661.464329] Lustre: Skipped 1 previous similar message [ 4661.604713] Lustre: lustre-MDT0001: new disk, initializing [ 4661.840529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4661.976899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4662.000440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4668.596575] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4677.416808] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4681.386810] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4689.277869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4689.881773] Lustre: lustre-OST0000: new disk, initializing [ 4689.888223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4689.896293] Lustre: 152249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4690.053971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.304544] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4698.825507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4698.843907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4698.908868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4706.092876] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4711.209768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4713.279816] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:00:41 (1779703241) [ 4953.057052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.060303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.072297] Lustre: Skipped 2 previous similar messages [ 4953.086387] Lustre: Skipped 2 previous similar messages [ 4958.176578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4958.184255] Lustre: Skipped 2 previous similar messages [ 4963.296219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.312590] Lustre: Skipped 2 previous similar messages [ 4966.369983] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.887352] Lustre: server umount lustre-MDT0000 complete [ 4968.416755] LustreError: 151495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.436510] LustreError: 151495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4971.417920] LustreError: 150497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703501 with bad export cookie 7530119285538002220 [ 4971.419469] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.441493] LustreError: 150497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4973.535814] LustreError: 150510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.536548] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.572410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4978.049535] Lustre: server umount lustre-MDT0001 complete [ 4989.159319] Lustre: server umount lustre-OST0000 complete [ 5002.726716] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 5015.610929] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5060.805927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.258039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5061.306315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5061.441059] Lustre: lustre-MDT0000: new disk, initializing [ 5061.598177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.627214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5067.091350] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5079.949047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.139158] Lustre: 156781:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5080.275988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5080.285693] Lustre: Skipped 1 previous similar message [ 5080.396844] Lustre: lustre-MDT0001: new disk, initializing [ 5080.543645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.614822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5080.632049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5085.502659] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5094.092756] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5098.772606] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5106.589294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5106.925077] Lustre: lustre-OST0000: new disk, initializing [ 5106.927458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5106.947197] Lustre: 158522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5107.067723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.984585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5113.998113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5114.172905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5114.353172] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5122.343181] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5127.603334] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5127.889613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5133.586848] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5133.904365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5139.945749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5139.948861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.958624] LustreError: Skipped 1 previous similar message [ 5139.958928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.958932] Lustre: Skipped 1 previous similar message [ 5140.015492] Lustre: Skipped 1 previous similar message [ 5143.139519] Lustre: server umount lustre-OST0000 complete [ 5150.196663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.205667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.216178] Lustre: Skipped 1 previous similar message [ 5150.239429] Lustre: Skipped 2 previous similar messages [ 5154.624737] Lustre: server umount lustre-MDT0000 complete [ 5155.308070] LustreError: 156791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.326263] LustreError: 156791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5158.935393] LustreError: 156773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703688 with bad export cookie 7530119285538161057 [ 5158.936948] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5158.946771] LustreError: 156773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5159.371818] Lustre: server umount lustre-MDT0001 complete [ 5170.487820] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:08:19 (1779703699) [ 5189.195418] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 5200.567899] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5239.808204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.245658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5240.333514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5240.560865] Lustre: lustre-MDT0000: new disk, initializing [ 5240.752546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.816318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5246.507609] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5258.377666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.601261] Lustre: 163117:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5258.809271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5264.061697] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5274.367481] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5278.705583] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5288.650736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5289.103780] Lustre: lustre-OST0000: new disk, initializing [ 5289.107453] Lustre: Skipped 1 previous similar message [ 5289.110388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5289.113460] Lustre: Skipped 2 previous similar messages [ 5289.117752] Lustre: 164871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5296.405348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5297.224294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5297.247366] Lustre: Skipped 1 previous similar message [ 5297.257120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5297.417964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5305.434052] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5318.118993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.119531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.133526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.144702] LustreError: Skipped 1 previous similar message [ 5321.799623] Lustre: server umount lustre-OST0000 complete [ 5328.353468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.378924] Lustre: Skipped 1 previous similar message [ 5333.276502] Lustre: server umount lustre-MDT0000 complete [ 5333.484713] LustreError: 165117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.768430] LustreError: 163111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779703868 with bad export cookie 7530119285538161939 [ 5338.783295] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.815905] Lustre: server umount lustre-MDT0001 complete [ 5350.714103] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:11:19 (1779703879) [ 5371.642247] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 5384.558167] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5425.894298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.286847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5426.354386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5426.458284] Lustre: lustre-MDT0000: new disk, initializing [ 5426.576484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.585680] Lustre: Skipped 2 previous similar messages [ 5426.615667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5431.018371] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5443.479570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.705903] Lustre: 169054:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5444.181821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.367149] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5458.860582] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.230654] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5470.321523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5470.796140] Lustre: 170801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.474275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5476.677332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5479.151629] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5488.147568] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5497.312430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.325954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.340429] Lustre: Skipped 1 previous similar message [ 5497.351673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.354377] Lustre: Skipped 3 previous similar messages [ 5503.502687] Lustre: server umount lustre-OST0000 complete [ 5510.628674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5510.639626] LustreError: Skipped 1 previous similar message [ 5514.942726] Lustre: server umount lustre-MDT0000 complete [ 5517.806165] LustreError: 169060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.828369] LustreError: 169060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5519.227833] LustreError: 169045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704049 with bad export cookie 7530119285538163024 [ 5519.233540] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5519.236693] LustreError: 169045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5519.640572] Lustre: server umount lustre-MDT0001 complete [ 5531.695129] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5534.948470] Key type lgssc unregistered [ 5535.505067] LNet: 172530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5535.508522] LNetError: 172530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5535.551911] LNet: Removed LNI 192.168.202.124@tcp [ 5536.878622] Key type .llcrypt unregistered [ 5536.881405] Key type ._llcrypt unregistered [ 5564.956331] Key type ._llcrypt registered [ 5564.958580] Key type .llcrypt registered [ 5565.093324] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5566.692932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5566.733488] alg: No test for adler32 (adler32-zlib) [ 5567.932656] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5568.219913] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5569.959239] Key type lgssc registered [ 5571.366816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.034619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.049404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.617505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.940358] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5591.012812] LustreError: 173928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.056355] LustreError: 173928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5596.142766] LustreError: 173929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.516737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5597.147356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5602.418709] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5609.028594] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5615.644099] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5624.293951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5624.784175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.814737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5632.108937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5638.912506] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5667.808555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5667.810173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.822893] LustreError: Skipped 1 previous similar message [ 5667.838260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5667.862950] Lustre: Skipped 1 previous similar message [ 5667.899403] Lustre: Skipped 1 previous similar message [ 5669.398931] Lustre: server umount lustre-OST0000 complete [ 5678.048069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.052278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5678.076265] Lustre: Skipped 1 previous similar message [ 5678.096660] Lustre: Skipped 1 previous similar message [ 5680.886967] Lustre: server umount lustre-MDT0000 complete [ 5683.174892] LustreError: 175039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.205960] LustreError: 175039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5685.991620] LustreError: 173908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704216 with bad export cookie 12797121640931624912 [ 5686.001754] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.006957] LustreError: 173908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5686.053217] LustreError: 173243:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 5686.439277] Lustre: server umount lustre-MDT0001 complete [ 5702.298911] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 5705.675938] Key type lgssc unregistered [ 5706.068140] LNet: 177129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5706.090152] LNetError: 177129:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5706.135947] LNet: Removed LNI 192.168.202.124@tcp [ 5707.214826] Key type .llcrypt unregistered [ 5707.218748] Key type ._llcrypt unregistered [ 5737.865765] Key type ._llcrypt registered [ 5737.869033] Key type .llcrypt registered [ 5737.988444] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 5739.132963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5739.257825] alg: No test for adler32 (adler32-zlib) [ 5740.322490] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5740.601646] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 5742.335174] Key type lgssc registered [ 5743.975639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5753.085893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5753.107425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.702932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.201329] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5764.077286] LustreError: 178535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.128096] LustreError: 178535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5769.190466] LustreError: 178536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.418566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.864416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5775.427730] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5782.284674] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5787.369762] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5794.257626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5794.592440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.727175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5800.930607] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5807.032577] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5814.878762] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 5815.265499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5815.280635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.318381] Lustre: Skipped 1 previous similar message [ 5819.925234] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 5819.936665] Lustre: Skipped 2 previous similar messages [ 5820.402401] LustreError: 180212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.625276] Lustre: server umount lustre-OST0000 complete [ 5827.124638] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ee148564800.ost_server_uuid 50 [ 5829.195710] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5845.611956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.626527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5845.645486] Lustre: Skipped 1 previous similar message [ 5845.661257] Lustre: Skipped 1 previous similar message [ 5851.117079] LustreError: 178535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.129330] LustreError: 178535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5851.162238] Lustre: server umount lustre-MDT0000 complete [ 5855.192953] LustreError: 178515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704385 with bad export cookie 12742171501192154797 [ 5855.206332] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.632141] Lustre: server umount lustre-MDT0001 complete [ 5865.684442] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:19:53 (1779704393) [ 5873.300256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.553494] LustreError: 182012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.604606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.715476] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5883.876841] LustreError: 182013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.892685] LustreError: 182013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5889.138778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.631100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5894.710102] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5901.176483] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.378723] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5913.481906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.839343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.962750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5921.079710] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5927.716592] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5936.190218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.398088] Lustre: lustre-OST0001: new disk, initializing [ 5936.401615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5936.408737] Lustre: 184514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5936.527081] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.308637] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 5946.568459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5946.575514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5946.803303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5951.998974] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5956.072713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5956.088966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.113426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.653775] Lustre: server umount lustre-OST0000 complete [ 5962.213608] LustreError: 184522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.235886] LustreError: 184522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5970.503074] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5970.750816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5976.470509] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5976.727294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5992.932727] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.933375] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.952909] LustreError: Skipped 1 previous similar message [ 5992.955751] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5992.955759] Lustre: Skipped 3 previous similar messages [ 5993.004898] Lustre: Skipped 2 previous similar messages [ 5998.052188] LustreError: 183606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.075404] LustreError: 183606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 5998.994822] Lustre: server umount lustre-OST0001 complete [ 6007.274301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6007.283969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.299149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.301756] Lustre: Skipped 3 previous similar messages [ 6009.795548] Lustre: server umount lustre-MDT0000 complete [ 6014.574875] LustreError: 181994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704544 with bad export cookie 12742171501192155868 [ 6014.585519] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.596637] LustreError: 181994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6015.348968] Lustre: server umount lustre-MDT0001 complete [ 6047.400585] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:22:55 (1779704575) [ 6056.524608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.871613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6056.898537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6057.223347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.915869] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6071.174536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.407978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6071.463774] Lustre: 187812:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6077.371548] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6083.776412] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6088.291487] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6096.363758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.589462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6105.068950] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6105.326919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6111.753252] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6119.404505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6119.558117] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6125.746981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6127.020817] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6134.041353] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6139.979219] Lustre: lustre-OST0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 6139.992345] Lustre: Skipped 2 previous similar messages [ 6141.409317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.410510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.420456] LustreError: Skipped 1 previous similar message [ 6141.453647] Lustre: Skipped 2 previous similar messages [ 6144.100343] Lustre: server umount lustre-OST0000 complete [ 6145.046403] LustreError: 189433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.078322] LustreError: 189433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6161.896524] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.918495] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.944843] Lustre: server umount lustre-OST0001 complete [ 6174.181324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.195973] Lustre: Skipped 6 previous similar messages [ 6180.112617] Lustre: server umount lustre-MDT0000 complete [ 6184.859631] LustreError: 187804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704714 with bad export cookie 12742171501192157275 [ 6184.871620] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.885302] LustreError: 187804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6185.545382] Lustre: server umount lustre-MDT0001 complete [ 6215.069427] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:25:43 (1779704743) [ 6234.014986] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 6244.336524] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6288.857242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.128521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6289.213220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6289.337551] Lustre: lustre-MDT0000: new disk, initializing [ 6289.412480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6289.419040] Lustre: Skipped 3 previous similar messages [ 6289.437806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6294.514559] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6307.009641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.168587] Lustre: 195426:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6307.254706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6307.261128] Lustre: Skipped 1 previous similar message [ 6307.369790] Lustre: lustre-MDT0001: new disk, initializing [ 6307.560383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6307.572964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6312.306574] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6322.591737] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6327.170534] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.144840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.513703] Lustre: lustre-OST0000: new disk, initializing [ 6336.528655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6336.539612] Lustre: 197169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6344.347127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6344.374403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6344.634975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6346.291454] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6354.964281] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6360.202531] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6360.406479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6366.578987] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6366.822398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6370.273865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6370.277960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.291753] LustreError: Skipped 3 previous similar messages [ 6370.300255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6370.309682] Lustre: Skipped 4 previous similar messages [ 6370.316979] Lustre: Skipped 2 previous similar messages [ 6374.859752] Lustre: server umount lustre-OST0000 complete [ 6386.835345] Lustre: server umount lustre-MDT0000 complete [ 6390.756062] LustreError: 195438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.832213] LustreError: 195438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6392.262143] LustreError: 195418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779704922 with bad export cookie 12742171501192159060 [ 6392.273725] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.274377] LustreError: 195418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6392.974068] Lustre: server umount lustre-MDT0001 complete [ 6404.629986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.569129] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6422.124292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.810545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6422.819881] Lustre: Skipped 3 previous similar messages [ 6427.936640] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6434.155434] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6438.577590] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6446.702374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.836919] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6461.840043] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6467.332352] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6467.518386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6473.261995] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6473.521191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6475.747638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.753783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.761710] Lustre: Skipped 2 previous similar messages [ 6481.961563] Lustre: server umount lustre-OST0000 complete [ 6489.494589] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6495.485834] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6528.964704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.303131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.311445] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.313110] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6531.328227] Lustre: Skipped 1 previous similar message [ 6535.231435] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6543.103368] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6545.565696] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6552.048173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6552.050852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6552.051356] LustreError: Skipped 1 previous similar message [ 6552.074964] Lustre: Skipped 10 previous similar messages [ 6557.011175] Lustre: server umount lustre-OST0000 complete [ 6569.124621] Lustre: server umount lustre-MDT0000 complete [ 6574.386963] LustreError: 199222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705104 with bad export cookie 12742171501192159942 [ 6574.418483] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6574.418988] LustreError: 199222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6575.019905] Lustre: server umount lustre-MDT0001 complete [ 6586.151579] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:31:54 (1779705114) [ 6594.731825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.069619] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6613.760808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.852971] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6626.544404] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6631.358416] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6641.109856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6649.742714] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6656.539540] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6662.370443] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6667.002237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6672.587278] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6672.880723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6680.617051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.876479] Lustre: lustre-OST0001: new disk, initializing [ 6680.880965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6680.886617] Lustre: 207722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6681.019166] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6681.028330] Lustre: Skipped 5 previous similar messages [ 6688.598549] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6689.385781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6689.410748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6689.585146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6698.165187] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6703.931289] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6704.154698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6709.327142] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6709.544452] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6713.329403] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.361067] Lustre: Skipped 5 previous similar messages [ 6726.111220] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6726.354080] Lustre: server umount lustre-OST0001 complete [ 6728.719773] LustreError: 207729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.750600] LustreError: 207729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6734.185292] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6734.479718] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6740.426279] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6740.657727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6771.483059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.337504] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6777.361919] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6777.368376] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6779.363912] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6786.747302] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6788.588170] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6792.673057] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.697765] LustreError: Skipped 2 previous similar messages [ 6796.660122] Lustre: server umount lustre-OST0001 complete [ 6817.761716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6817.772495] Lustre: Skipped 11 previous similar messages [ 6822.667739] Lustre: server umount lustre-OST0000 complete [ 6841.311188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6841.721498] Lustre: server umount lustre-MDT0000 complete [ 6845.721884] LustreError: 204678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705375 with bad export cookie 12742171501192160908 [ 6845.729267] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.749808] LustreError: 204678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6860.259658] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6860.775800] Lustre: server umount lustre-MDT0001 complete [ 6889.664528] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:36:58 (1779705418) [ 6897.410035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.661613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6897.711277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6903.511280] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6913.686424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.896454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6913.956939] Lustre: 212678:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6919.873673] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6927.471591] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6932.680381] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6941.579083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.967055] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6950.576596] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6957.332826] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6968.708503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.778175] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6976.068624] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6982.009256] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6983.640269] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6989.145347] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6989.440930] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6995.161663] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6995.386702] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6999.909562] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ee1449a9800.ost_server_uuid 50 [ 7002.150359] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ee1449a9800.ost_server_uuid in IDLE state after 0 sec [ 7003.532420] Lustre: Permanently deactivating lustre-OST0001 [ 7003.568233] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7007.857402] Lustre: Permanently reactivating lustre-OST0001 [ 7007.875286] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7007.878894] Lustre: Skipped 2 previous similar messages [ 7013.346373] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.365796] Lustre: Skipped 6 previous similar messages [ 7019.639367] Lustre: server umount lustre-OST0001 complete [ 7032.058559] Lustre: server umount lustre-OST0000 complete [ 7042.964206] Lustre: server umount lustre-MDT0000 complete [ 7047.605192] LustreError: 214314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705577 with bad export cookie 12742171501192162196 [ 7047.621860] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7047.630691] LustreError: 214314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7048.244583] Lustre: server umount lustre-MDT0001 complete [ 7078.138229] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:40:06 (1779705606) [ 7086.716742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.192476] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7087.233799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7087.242749] Lustre: Skipped 2 previous similar messages [ 7092.774410] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7103.471128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.540971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7103.575191] Lustre: 218790:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.246375] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7115.811770] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7119.648453] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7127.203337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7127.416761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7127.432616] Lustre: Permanently deactivating lustre-OST0000 [ 7127.442949] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7134.171066] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7136.910097] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7140.586505] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7148.626420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.683528] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7155.018300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7155.445855] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7161.648416] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7163.433603] Lustre: Permanently reactivating lustre-OST0000 [ 7163.446187] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7163.451709] Lustre: Skipped 2 previous similar messages [ 7172.626302] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 000000006ff0048c, cur 1779705702 deadline 1779705696 last 1779705666 [ 7173.273075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7181.283056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7181.311028] LustreError: Skipped 4 previous similar messages [ 7191.007930] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7191.128797] Lustre: server umount lustre-OST0001 complete [ 7201.868614] Lustre: server umount lustre-OST0000 complete [ 7213.401770] Lustre: server umount lustre-MDT0000 complete [ 7217.509274] LustreError: 218784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705747 with bad export cookie 12742171501192163862 [ 7217.520070] LustreError: 218784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7217.520619] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.909428] Lustre: server umount lustre-MDT0001 complete [ 7231.616544] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7235.357873] Key type lgssc unregistered [ 7235.795475] LNet: 223199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7235.802679] LNetError: 223199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7236.862504] LNet: Removed LNI 192.168.202.124@tcp [ 7237.766597] Key type .llcrypt unregistered [ 7237.769131] Key type ._llcrypt unregistered [ 7262.507754] Key type ._llcrypt registered [ 7262.510338] Key type .llcrypt registered [ 7262.613975] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7264.015392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7264.052959] alg: No test for adler32 (adler32-zlib) [ 7265.093585] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7265.306684] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7266.959172] Key type lgssc registered [ 7268.245318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.187762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7281.210504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.746059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.920403] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7292.896521] LustreError: 224651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.925284] LustreError: 224651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7297.219955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.468487] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7297.530099] Lustre: 224640:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7297.549822] Lustre: Permanently deactivating lustre-MDT0001 [ 7297.584816] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7298.089590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7303.764267] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7306.302907] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7310.885208] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7315.326162] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7323.259250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7323.731582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.855695] LustreError: 226470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.883789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7330.691672] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7333.354902] LustreError: 226470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.182683] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7338.470545] LustreError: 226267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.571871] LustreError: 226264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.604763] LustreError: 226264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7346.464296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.709982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.817692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7351.841639] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7354.040404] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7361.478395] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7365.901148] Lustre: Permanently reactivating lustre-MDT0001 [ 7365.921278] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7365.924757] Lustre: Skipped 2 previous similar messages [ 7370.821944] Lustre: lustre-MDT0001: Client e89ffc69-53cb-4ddc-a484-b33cb274f7df (at 192.168.202.24@tcp) reconnecting [ 7373.445150] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.459638] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7373.471570] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7373.488531] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7376.063543] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7378.145899] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7385.860934] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7386.091681] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7391.666306] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7391.970594] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7395.006600] Lustre: Permanently deactivating lustre-MDT0001 [ 7395.024606] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7395.029358] Lustre: Skipped 2 previous similar messages [ 7405.226648] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7415.777762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7415.797228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.838338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.505693] Lustre: server umount lustre-MDT0000 complete [ 7419.360392] LustreError: 224652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.365244] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.389697] Lustre: Skipped 2 previous similar messages [ 7422.938122] LustreError: 228556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779705952 with bad export cookie 1192706196543807307 [ 7422.955603] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7422.975764] LustreError: 228556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7423.674935] Lustre: server umount lustre-MDT0001 complete [ 7431.132180] Lustre: server umount lustre-OST0000 complete [ 7444.839806] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:46:13 (1779705973) [ 7458.157064] Lustre: server umount lustre-OST0001 complete [ 7465.490337] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 7477.120348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7521.001208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.421119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7521.447568] Lustre: Skipped 2 previous similar messages [ 7521.511126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7521.659482] Lustre: lustre-MDT0000: new disk, initializing [ 7521.795706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.823305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7526.428300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7538.964913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.162093] Lustre: 232348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7539.279931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7539.284434] Lustre: Skipped 1 previous similar message [ 7539.392830] Lustre: lustre-MDT0001: new disk, initializing [ 7539.515951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.597724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7539.608975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7544.558549] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7552.686966] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7556.905497] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7564.680661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.969340] Lustre: lustre-OST0000: new disk, initializing [ 7564.973279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7564.980283] Lustre: 234090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7565.114683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.682956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7571.703341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7571.869795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7573.610530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7582.490720] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7585.113097] LustreError: 232355:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7595.130391] LustreError: 232355:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7595.176789] LustreError: 232354:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7595.350526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7595.532253] Lustre: lustre-OST0001: new disk, initializing [ 7595.535993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7595.546176] Lustre: 235094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7603.761482] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7604.948698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7604.960242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7605.053357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7605.225611] LustreError: 232354:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7613.374292] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7616.230984] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7620.590450] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7620.599362] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.600186] LustreError: Skipped 1 previous similar message [ 7620.608551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7620.623942] Lustre: Skipped 1 previous similar message [ 7620.643383] Lustre: Skipped 1 previous similar message [ 7624.989082] Lustre: server umount lustre-OST0001 complete [ 7625.705995] LustreError: 234363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.737605] LustreError: 234363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7646.180152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.199583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.213663] Lustre: Skipped 1 previous similar message [ 7650.383608] Lustre: server umount lustre-OST0000 complete [ 7656.417234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.448622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.463563] Lustre: Skipped 2 previous similar messages [ 7656.476800] Lustre: Skipped 1 previous similar message [ 7661.542310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7661.554471] Lustre: Skipped 1 previous similar message [ 7670.239317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7670.700879] Lustre: server umount lustre-MDT0000 complete [ 7671.789171] LustreError: 232354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.810665] LustreError: 232354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7675.517463] LustreError: 232339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706205 with bad export cookie 1192706196543809904 [ 7675.530773] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7675.538884] LustreError: 232339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7682.197113] Lustre: server umount lustre-MDT0001 complete [ 7696.752617] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7700.667683] Key type lgssc unregistered [ 7700.985969] LNet: 237002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7700.992170] LNetError: 237002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7702.063163] LNet: Removed LNI 192.168.202.124@tcp [ 7703.172693] Key type .llcrypt unregistered [ 7703.177877] Key type ._llcrypt unregistered [ 7751.934490] Key type ._llcrypt registered [ 7751.936962] Key type .llcrypt registered [ 7752.092071] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7753.489387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7753.652403] alg: No test for adler32 (adler32-zlib) [ 7754.817102] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7755.173256] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7756.944222] Key type lgssc registered [ 7758.723754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7768.595968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7768.628582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.127243] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7770.160514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7770.399987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.265088] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7785.067320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7785.154726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7785.195482] Lustre: 239197:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7785.544240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7790.750834] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7797.476067] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7802.223807] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7810.426989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7810.629246] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7810.839051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.835768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7817.425348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7823.742217] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7883.232342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.248344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.266412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.755436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.768248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.788738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7885.997432] Lustre: server umount lustre-OST0000 complete [ 7891.695133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7907.827135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.377437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.700178] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7921.829290] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7923.752907] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7923.773269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7923.790902] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7964.663100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.676750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.683100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7966.695994] Lustre: server umount lustre-OST0000 complete [ 7974.892302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7974.894142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.919498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.939955] Lustre: Skipped 2 previous similar messages [ 7974.962166] Lustre: Skipped 2 previous similar messages [ 7977.007311] Lustre: server umount lustre-MDT0000 complete [ 7982.111637] LustreError: 239188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706512 with bad export cookie 15423326107696819126 [ 7982.129735] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7982.704567] Lustre: server umount lustre-MDT0001 complete [ 7996.886367] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8000.354662] Key type lgssc unregistered [ 8000.763447] LNet: 244147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8000.770829] LNetError: 244147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8000.784466] LNet: Removed LNI 192.168.202.124@tcp [ 8002.092439] Key type .llcrypt unregistered [ 8002.094427] Key type ._llcrypt unregistered [ 8030.980742] Key type ._llcrypt registered [ 8030.985631] Key type .llcrypt registered [ 8031.157067] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8033.035539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8033.052605] alg: No test for adler32 (adler32-zlib) [ 8034.064388] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8034.339513] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8036.031602] Key type lgssc registered [ 8037.117356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8046.084782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8046.104763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.800804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.818566] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8057.316031] LustreError: 245551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.334717] LustreError: 245551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8062.443111] LustreError: 245550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8063.319136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.846469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8068.261159] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8075.744452] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8080.394290] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8089.089705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.667077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.757218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8097.150977] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8103.654334] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8138.722732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.728825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.744664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.349443] Lustre: server umount lustre-OST0000 complete [ 8145.909828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8145.917107] LustreError: Skipped 1 previous similar message [ 8145.920721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8145.932397] Lustre: Skipped 1 previous similar message [ 8145.943567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8145.946873] Lustre: Skipped 1 previous similar message [ 8148.962537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8148.966618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.007118] Lustre: Skipped 1 previous similar message [ 8154.084128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.111718] Lustre: Skipped 1 previous similar message [ 8158.178534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8159.199139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8159.419155] Lustre: server umount lustre-MDT0000 complete [ 8163.315215] LustreError: 245551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.364494] LustreError: 245551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8164.342497] LustreError: 245531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706694 with bad export cookie 13179269625205367642 [ 8164.370755] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8164.792604] Lustre: server umount lustre-MDT0001 complete [ 8178.506585] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8181.498860] Key type lgssc unregistered [ 8181.871752] LNet: 249504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8181.880912] LNetError: 249504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8181.912574] LNet: Removed LNI 192.168.202.124@tcp [ 8182.745482] Key type .llcrypt unregistered [ 8182.748305] Key type ._llcrypt unregistered [ 8203.114828] Key type ._llcrypt registered [ 8203.116905] Key type .llcrypt registered [ 8203.216425] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8204.556600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8204.638532] alg: No test for adler32 (adler32-zlib) [ 8205.968387] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8206.406233] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8208.119221] Key type lgssc registered [ 8209.741076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8219.203239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.238938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.868888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.594887] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8230.379411] LustreError: 250710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.414628] LustreError: 250710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8235.489896] LustreError: 250711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8237.290180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.957384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8243.268149] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8248.976792] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8253.590105] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8262.244570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.711982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.809808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8270.685336] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8276.888945] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8291.299321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.302750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8291.315784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.424083] LustreError: 252305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.438235] LustreError: 252305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8296.533604] Lustre: server umount lustre-OST0000 complete [ 8301.541540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8301.545791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8301.566848] Lustre: Skipped 2 previous similar messages [ 8301.599985] Lustre: Skipped 2 previous similar messages [ 8306.656830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8306.663827] Lustre: Skipped 1 previous similar message [ 8311.780070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8311.788519] Lustre: Skipped 1 previous similar message [ 8315.359135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8315.610399] Lustre: server umount lustre-MDT0000 complete [ 8316.899668] LustreError: 250711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.915978] LustreError: 250711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8319.782322] LustreError: 250692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706849 with bad export cookie 483521691156321462 [ 8319.793390] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.232529] Lustre: server umount lustre-MDT0001 complete [ 8333.095746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8336.184161] Key type lgssc unregistered [ 8336.560309] LNet: 254047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8336.567767] LNetError: 254047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8336.610061] LNet: Removed LNI 192.168.202.124@tcp [ 8337.861257] Key type .llcrypt unregistered [ 8337.867150] Key type ._llcrypt unregistered [ 8368.240597] Key type ._llcrypt registered [ 8368.246971] Key type .llcrypt registered [ 8368.405894] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8370.171488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8370.185648] alg: No test for adler32 (adler32-zlib) [ 8371.226134] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8371.461547] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8373.183267] Key type lgssc registered [ 8374.788676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8382.817412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8382.847613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8384.385789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.405260] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8394.725189] LustreError: 255449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.746680] LustreError: 255449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8399.123426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8399.581835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8405.400278] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8412.201891] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8416.778916] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8426.315802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8426.824167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.888733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8434.468973] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8441.308831] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8475.108710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.110972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8475.132185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8479.230226] Lustre: server umount lustre-OST0000 complete [ 8485.346760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.362707] Lustre: Skipped 1 previous similar message [ 8485.369504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.381787] Lustre: Skipped 2 previous similar messages [ 8490.466842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.477171] Lustre: Skipped 1 previous similar message [ 8491.033073] Lustre: server umount lustre-MDT0000 complete [ 8495.584384] LustreError: 255446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.606769] LustreError: 255446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8495.904848] LustreError: 255429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707025 with bad export cookie 8667760185079245216 [ 8495.913658] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.919731] LustreError: 255429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8496.574590] Lustre: server umount lustre-MDT0001 complete [ 8510.439194] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8513.793928] Key type lgssc unregistered [ 8514.243766] LNet: 259403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8514.260371] LNetError: 259403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8514.272360] LNet: Removed LNI 192.168.202.124@tcp [ 8515.478431] Key type .llcrypt unregistered [ 8515.482099] Key type ._llcrypt unregistered [ 8536.968249] Key type ._llcrypt registered [ 8536.976239] Key type .llcrypt registered [ 8537.127657] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8538.508907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8538.738594] alg: No test for adler32 (adler32-zlib) [ 8539.808766] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8540.003638] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8541.731096] Key type lgssc registered [ 8543.144230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8553.139598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8553.157162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.097879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.027927] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8565.227379] LustreError: 260641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.259071] LustreError: 260641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8570.346773] LustreError: 260643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8571.419875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.923808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8577.451223] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8583.602912] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8588.724176] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8596.584222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8596.995971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.043233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8604.328370] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8610.762029] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8626.658732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.661920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.680236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.702118] Lustre: Skipped 1 previous similar message [ 8631.783636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8631.796028] Lustre: Skipped 1 previous similar message [ 8632.977559] Lustre: server umount lustre-OST0000 complete [ 8638.436239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8638.451167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.487322] Lustre: Skipped 1 previous similar message [ 8638.501681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.017679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.020252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.034046] Lustre: Skipped 1 previous similar message [ 8647.139548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.146813] Lustre: Skipped 1 previous similar message [ 8652.255481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8652.646723] Lustre: server umount lustre-MDT0000 complete [ 8656.359105] LustreError: 260654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.374791] LustreError: 260654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8657.395743] LustreError: 262962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707187 with bad export cookie 10050172186007397262 [ 8657.409458] LustreError: 262962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8657.409992] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8657.945658] Lustre: server umount lustre-MDT0001 complete [ 8671.011656] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8674.260124] Key type lgssc unregistered [ 8674.652593] LNet: 263989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8674.665797] LNetError: 263989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8674.683898] LNet: Removed LNI 192.168.202.124@tcp [ 8675.503532] Key type .llcrypt unregistered [ 8675.505518] Key type ._llcrypt unregistered [ 8688.754507] Key type ._llcrypt registered [ 8688.757651] Key type .llcrypt registered [ 8688.937061] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8708.995583] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8725.497993] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8726.668756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8726.751121] alg: No test for adler32 (adler32-zlib) [ 8728.003222] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 8728.367123] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8730.167218] Key type lgssc registered [ 8731.795337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8773.297570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8773.334660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.762817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8774.825267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8774.925647] Lustre: lustre-MDT0000: new disk, initializing [ 8775.049888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.083646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8780.684973] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8794.127585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8794.228915] Lustre: 267503:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8794.285551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8794.290754] Lustre: Skipped 1 previous similar message [ 8794.368342] Lustre: lustre-MDT0001: new disk, initializing [ 8794.476658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8794.566225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8794.588699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8800.836058] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8810.647862] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8815.401720] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8824.900322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8825.418913] Lustre: lustre-OST0000: new disk, initializing [ 8825.422863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8825.428714] Lustre: 269243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8825.537398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.756927] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8834.165623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8834.172542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8834.305313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8844.102792] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8850.023308] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8850.287802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8856.829718] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8857.144129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8860.141112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8860.143082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.151132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.195266] Lustre: Skipped 1 previous similar message [ 8865.250427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.259970] Lustre: Skipped 2 previous similar messages [ 8866.056946] Lustre: server umount lustre-OST0000 complete [ 8875.490731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.493939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8875.523669] Lustre: Skipped 1 previous similar message [ 8875.561290] Lustre: Skipped 1 previous similar message [ 8878.549650] Lustre: server umount lustre-MDT0000 complete [ 8884.169462] LustreError: 267493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707414 with bad export cookie 4477549772108247738 [ 8884.198360] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8884.895901] Lustre: server umount lustre-MDT0001 complete [ 8896.734424] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:10:25 (1779707425) [ 8905.160296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.641720] LustreError: 271511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.666071] LustreError: 271511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8905.728553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.819163] LustreError: 271512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.963512] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8915.947262] LustreError: 271511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.061149] LustreError: 271512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.092508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.622196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8927.703172] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8935.122766] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8941.112659] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8949.333155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.667171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.674284] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8963.469349] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8979.734555] Lustre: server umount lustre-OST0000 complete [ 8986.592427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.614035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.624598] Lustre: Skipped 1 previous similar message [ 8991.725239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.731712] Lustre: Skipped 1 previous similar message [ 8997.791169] Lustre: 265727:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779707511/real 1779707511] req@00000000fd513076 x1866158317843200/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779707527 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8997.826513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.843586] Lustre: Skipped 1 previous similar message [ 8998.367151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8998.632509] Lustre: server umount lustre-MDT0000 complete [ 9002.975710] Lustre: 265726:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779707516/real 1779707516] req@00000000ab1e463d x1866158317844992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779707532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9002.989604] LustreError: 271507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.038877] Lustre: 265726:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9003.112146] LustreError: 271507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9003.286758] LustreError: 273130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707533 with bad export cookie 4477549772108248620 [ 9003.291902] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9003.299764] LustreError: 273130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9003.780299] Lustre: server umount lustre-MDT0001 complete [ 9016.587471] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 9020.440933] Key type lgssc unregistered [ 9020.740690] LNet: 274713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9020.749529] LNetError: 274713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9020.764897] LNet: Removed LNI 192.168.202.124@tcp [ 9021.544584] Key type .llcrypt unregistered [ 9021.547464] Key type ._llcrypt unregistered [ 9052.058471] Key type ._llcrypt registered [ 9052.070264] Key type .llcrypt registered [ 9052.245025] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9069.661268] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9070.856284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9070.921368] alg: No test for adler32 (adler32-zlib) [ 9072.268575] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 9072.540722] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 9074.231615] Key type lgssc registered [ 9076.036642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9128.026803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9128.067174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.432444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9129.460837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9129.528156] Lustre: lustre-MDT0000: new disk, initializing [ 9129.608835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9129.649845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9134.372681] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9147.064537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.204131] Lustre: 278513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9147.284784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9147.288381] Lustre: Skipped 1 previous similar message [ 9147.375808] Lustre: lustre-MDT0001: new disk, initializing [ 9147.499314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.590821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9147.609405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9152.559978] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9161.367611] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9165.988030] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9173.470182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9173.757293] Lustre: lustre-OST2710: new disk, initializing [ 9173.767660] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9173.777587] Lustre: 280254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9173.906086] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9180.016519] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9182.345174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9182.360306] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9182.464540] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9190.116895] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9198.836439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9199.066664] Lustre: lustre-OST03e8: new disk, initializing [ 9199.071620] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9199.078040] Lustre: 281205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9199.202107] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9204.834675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9204.864732] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9205.032564] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9207.533694] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9213.920476] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9221.886071] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9222.223083] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9228.776929] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9229.018896] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9233.893192] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9234.174542] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9239.635310] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9239.834821] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9246.178050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9246.188535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9246.203410] Lustre: Skipped 2 previous similar messages [ 9246.215021] Lustre: Skipped 1 previous similar message [ 9251.305573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.320430] Lustre: Skipped 5 previous similar messages [ 9252.769653] Lustre: server umount lustre-MDT0000 complete [ 9258.228830] LustreError: 278504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707788 with bad export cookie 7544977880175468618 [ 9258.237537] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9258.248510] LustreError: 278504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9258.947757] Lustre: server umount lustre-MDT0001 complete [ 9275.120129] Lustre: server umount lustre-OST2710 complete [ 9277.791123] Lustre: 276310:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779707791/real 1779707791] req@0000000080e549fd x1866158678546816/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779707807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9277.824229] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9277.847916] Lustre: Skipped 1 previous similar message [ 9279.701293] Lustre: server umount lustre-OST03e8 complete [ 9301.649293] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9313.782487] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9357.897804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9358.319873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9358.374129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9358.535557] Lustre: lustre-MDT0000: new disk, initializing [ 9358.684950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9358.708303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9364.366563] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9376.806673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.040476] Lustre: 286244:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9377.151173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9377.158977] Lustre: Skipped 1 previous similar message [ 9377.370742] Lustre: lustre-MDT0001: new disk, initializing [ 9377.656478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9377.756710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9377.783827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9383.716697] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9393.208454] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9397.512971] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9405.472510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9405.663412] Lustre: lustre-OST0000: new disk, initializing [ 9405.666545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9405.672026] Lustre: 287988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9405.749425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9412.194650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9412.212546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9412.335668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9412.487146] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9421.185319] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9426.742806] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9427.090742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9431.906930] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9432.298614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9438.176361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9438.182702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.190317] LustreError: Skipped 1 previous similar message [ 9438.196364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9438.204522] Lustre: Skipped 1 previous similar message [ 9438.223186] Lustre: Skipped 1 previous similar message [ 9440.589096] Lustre: server umount lustre-OST0000 complete [ 9448.417856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.444663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9448.453783] Lustre: Skipped 1 previous similar message [ 9451.335097] Lustre: server umount lustre-MDT0000 complete [ 9453.540287] LustreError: 286255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9453.576349] LustreError: 286255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9455.700722] LustreError: 286236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707985 with bad export cookie 7544977880175470298 [ 9455.706501] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9455.725057] LustreError: 286236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9456.369669] Lustre: server umount lustre-MDT0001 complete [ 9466.453574] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:19:55 (1779707995) [ 9467.909372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9470.362961] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:19:58 (1779707998) [ 9474.784464] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9509.161657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.586427] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9509.619800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9509.890492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.491274] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9526.131749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.283906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9526.331924] Lustre: 291684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9526.504119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9531.888590] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9555.454671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.885321] Lustre: Denying initial registration attempt from nid 192.168.202.124@tcp, specified as failover [ 9555.902508] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9555.913106] LustreError: 293104:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9555.931497] LustreError: 293104:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9555.947154] LustreError: 293104:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9556.037777] Lustre: server umount lustre-OST0000 complete [ 9556.041219] LustreError: 293104:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9567.713294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9567.723569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.732085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.752101] Lustre: Skipped 2 previous similar messages [ 9567.766967] Lustre: Skipped 1 previous similar message [ 9571.461793] Lustre: server umount lustre-MDT0000 complete [ 9572.853195] LustreError: 291696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.882949] LustreError: 291696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9576.699088] LustreError: 291675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708106 with bad export cookie 7544977880175471180 [ 9576.715606] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9576.721494] LustreError: 291675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9577.461753] Lustre: server umount lustre-MDT0001 complete [ 9590.264157] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 9593.632683] Key type lgssc unregistered [ 9594.239493] LNet: 294208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9594.262564] LNetError: 294208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9594.307391] LNet: Removed LNI 192.168.202.124@tcp [ 9595.564527] Key type .llcrypt unregistered [ 9595.571403] Key type ._llcrypt unregistered [ 9611.154853] Key type ._llcrypt registered [ 9611.157422] Key type .llcrypt registered [ 9611.265137] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9613.349434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9613.400721] alg: No test for adler32 (adler32-zlib) [ 9615.051844] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 9615.602741] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 9617.479271] Key type lgssc registered [ 9619.344134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9673.532532] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9684.976066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9684.995812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.504848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9686.602877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9686.815430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.064923] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9702.568970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.661429] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9702.733073] Lustre: 296943:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9703.108712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.329857] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9715.244580] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9721.332906] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9743.284797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9743.630699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9743.880344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.556857] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9761.135356] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9767.393598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9767.395911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.426531] LustreError: Skipped 1 previous similar message [ 9767.428143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9767.476327] Lustre: Skipped 1 previous similar message [ 9770.122620] Lustre: server umount lustre-OST0000 complete [ 9777.640162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.647765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9777.652078] Lustre: Skipped 1 previous similar message [ 9777.659599] Lustre: Skipped 2 previous similar messages [ 9782.755642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9782.762334] Lustre: Skipped 1 previous similar message [ 9783.201389] Lustre: server umount lustre-MDT0000 complete [ 9787.157213] LustreError: 296936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708317 with bad export cookie 4626851871572612670 [ 9787.161584] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9787.165189] LustreError: 296936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9787.579678] Lustre: server umount lustre-MDT0001 complete [ 9799.541459] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 9802.895326] Key type lgssc unregistered [ 9803.289754] LNet: 300163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9803.302521] LNetError: 300163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9803.319246] LNet: Removed LNI 192.168.202.124@tcp [ 9804.397670] Key type .llcrypt unregistered [ 9804.400095] Key type ._llcrypt unregistered [ 9833.450678] Key type ._llcrypt registered [ 9833.458571] Key type .llcrypt registered [ 9833.619415] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9834.827420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9834.941043] alg: No test for adler32 (adler32-zlib) [ 9836.083720] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 9836.384383] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 9838.087236] Key type lgssc registered [ 9839.457478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9848.498543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9848.520653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9850.113736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.675749] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9860.582105] LustreError: 301567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.610488] LustreError: 301567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9863.996337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.576618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9870.229814] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9877.432416] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9882.456605] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9892.184957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9892.762347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.195706] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9908.522313] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9919.460090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.468297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9919.485072] Lustre: Skipped 1 previous similar message [ 9920.578351] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [ 9920.585016] Lustre: Skipped 2 previous similar messages [ 9924.530779] Lustre: server umount lustre-MDT0000 complete [ 9924.589789] LustreError: 301567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.628194] LustreError: 301567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9925.675228] LustreError: 301563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.595337] LustreError: 301549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708459 with bad export cookie 8897474721478283634 [ 9929.628129] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.696482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.698640] LustreError: 302679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.716983] Lustre: Skipped 1 previous similar message [ 9929.723145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9934.826583] LustreError: 301564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.829759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9934.846024] LustreError: 301564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9934.863376] Lustre: Skipped 1 previous similar message [ 9936.227582] Lustre: server umount lustre-MDT0001 complete [ 9944.396208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9955.779839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.755657] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7b7a2e1e4bbb2d79 [ 9974.778050] Lustre: MGC192.168.202.124@tcp: Connection restored to 0@lo (at 0@lo) [ 9975.089756] LustreError: 303174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.104734] LustreError: 303174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9975.209447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9975.268899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9980.640236] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9990.638461] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9991.017749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.197638] LustreError: 305254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.227736] LustreError: 305254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 9991.651771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9996.814931] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9998.881498] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10005.839597] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10007.703511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10011.584176] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10013.358437] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10021.344906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10021.353441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10021.374565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10021.378936] Lustre: Skipped 1 previous similar message [10024.593150] Lustre: server umount lustre-OST0000 complete [10032.609092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.610162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10032.623086] Lustre: Skipped 2 previous similar messages [10032.635959] Lustre: Skipped 3 previous similar messages [10043.359152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10043.643941] Lustre: server umount lustre-MDT0000 complete [10047.111897] LustreError: 301549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708577 with bad export cookie 8897474721478290809 [10047.115675] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10047.120671] LustreError: 301549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10047.442332] Lustre: server umount lustre-MDT0001 complete [10058.378397] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [10061.366686] Key type lgssc unregistered [10061.685947] LNet: 307256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10061.698322] LNetError: 307256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10061.711913] LNet: Removed LNI 192.168.202.124@tcp [10062.713648] Key type .llcrypt unregistered [10062.715305] Key type ._llcrypt unregistered [10088.236353] Key type ._llcrypt registered [10088.238333] Key type .llcrypt registered [10088.350458] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10089.618909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10089.670157] alg: No test for adler32 (adler32-zlib) [10090.846380] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [10091.115890] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [10092.767534] Key type lgssc registered [10094.051731] Lustre: Echo OBD driver; http://www.lustre.org/ [10105.446190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10105.502988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.342306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.322365] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10117.612770] LustreError: 308656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.651971] LustreError: 308656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10122.721566] LustreError: 308657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.608034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.216703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.415292] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10136.801555] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10142.154396] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10149.097102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.105887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.121548] Lustre: Skipped 1 previous similar message [10149.146793] Lustre: Skipped 1 previous similar message [10152.477619] Lustre: server umount lustre-MDT0000 complete [10154.473368] LustreError: 308652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.512223] LustreError: 308652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10156.512526] LustreError: 310048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708686 with bad export cookie 6070756456499961990 [10156.514514] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10156.519584] LustreError: 310048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10156.866682] Lustre: server umount lustre-MDT0001 complete [10173.664286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.907892] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10173.984858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10174.285492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10179.657680] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10192.159406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.244822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10192.289988] Lustre: 311018:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10192.603149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.613292] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10204.638114] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10208.386951] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10219.126425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10219.290740] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10219.308800] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10219.317424] LustreError: 312680:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [10219.324523] LustreError: 312680:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [10219.339529] LustreError: 312680:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10219.429853] Lustre: server umount lustre-OST0000 complete [10219.437910] LustreError: 312680:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10230.764499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10230.933916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10231.114965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10237.606734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10237.723542] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10243.074575] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10251.199555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10251.286432] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10251.359369] Lustre: lustre-OST0001: new disk, initializing [10251.362606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10251.370450] Lustre: 314108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10251.482558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10258.053771] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10261.119860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10261.131943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10261.248987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10264.572154] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10271.717500] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10271.775553] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10271.799914] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10273.551978] Lustre: server umount lustre-OST0001 complete [10276.845090] LustreError: 313148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.858868] LustreError: 313148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10278.368683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10278.376543] LustreError: Skipped 1 previous similar message [10278.380829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.400934] Lustre: Skipped 1 previous similar message [10278.407601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10278.413223] Lustre: Skipped 1 previous similar message [10281.956983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10281.961075] LustreError: 313148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.969142] Lustre: Skipped 2 previous similar messages [10282.011370] LustreError: 313148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10284.653830] Lustre: server umount lustre-OST0000 complete [10290.145279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10290.150949] LustreError: Skipped 1 previous similar message [10290.155456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10290.163335] Lustre: Skipped 1 previous similar message [10290.167836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.142083] Lustre: server umount lustre-MDT0000 complete [10297.313717] LustreError: 311032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.326833] LustreError: 311032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10298.853930] LustreError: 311013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708828 with bad export cookie 6070756456499962543 [10298.870938] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.874040] LustreError: 311013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10299.147771] Lustre: server umount lustre-MDT0001 complete [10310.874375] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [10314.786882] Key type lgssc unregistered [10315.307831] LNet: 315936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10315.333864] LNetError: 315936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10315.388446] LNet: Removed LNI 192.168.202.124@tcp [10316.878889] Key type .llcrypt unregistered [10316.883049] Key type ._llcrypt unregistered [10401.884594] Key type ._llcrypt registered [10401.889244] Key type .llcrypt registered [10402.019839] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [10419.799448] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10421.555027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10421.600646] alg: No test for adler32 (adler32-zlib) [10422.746892] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [10423.135529] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [10424.855227] Key type lgssc registered [10426.245687] Lustre: Echo OBD driver; http://www.lustre.org/ [10467.169274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10467.236679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.620975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10468.677298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10468.788622] Lustre: lustre-MDT0000: new disk, initializing [10468.916800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.940285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10473.752123] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10485.777136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.931739] Lustre: 320841:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10485.978722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10485.982167] Lustre: Skipped 1 previous similar message [10486.042864] Lustre: lustre-MDT0001: new disk, initializing [10486.132544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10486.196400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10486.211923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10491.685546] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10499.868389] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10503.551778] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10509.616509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.780266] Lustre: lustre-OST0000: new disk, initializing [10509.784980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10509.789227] Lustre: 322582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10509.862976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10517.142095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10517.168280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10517.342545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10517.826113] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10526.732069] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10531.891710] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10532.070438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10537.025906] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10537.210238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10543.076789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.089537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10543.112502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10545.890958] Lustre: server umount lustre-OST0000 complete [10552.804936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10552.813233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.831809] Lustre: Skipped 1 previous similar message [10552.837568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10552.840317] Lustre: Skipped 1 previous similar message [10557.850282] Lustre: server umount lustre-MDT0000 complete [10562.273219] LustreError: 320833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709092 with bad export cookie 2239247029873773945 [10562.294689] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10563.112351] Lustre: server umount lustre-MDT0001 complete [10573.568509] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:38:21 (1779709101) [10583.196039] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:38:31 (1779709111) [10594.311627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.781529] LustreError: 325194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10594.810199] LustreError: 325194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10594.866243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.846812] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10599.905349] LustreError: 325195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.002916] LustreError: 325194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.049579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.295078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10612.732041] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10619.060959] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10622.893858] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10630.525116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.018271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10640.425789] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10647.188856] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10656.276766] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10658.033864] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10660.001774] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10662.228361] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10664.460824] Lustre: Failing over lustre-MDT0000 [10664.810171] Lustre: server umount lustre-MDT0000 complete [10664.936788] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10664.941382] LustreError: 325195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.957365] Lustre: Skipped 3 previous similar messages [10664.986343] LustreError: 325195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10670.049733] LustreError: 326173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10670.068953] LustreError: 326173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10680.291967] LustreError: 327345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.320047] LustreError: 327345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10686.398957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.431096] Lustre: 319066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709200/real 1779709200] req@00000000c95aa9a4 x1866160095172992/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779709216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10686.461384] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.674612] LustreError: 325194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.682965] LustreError: 325194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [10697.119983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10697.205182] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10698.800072] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10701.499348] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10702.309881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10702.341866] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10702.392860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10709.258065] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10711.123772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10714.869590] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10717.142601] Lustre: Failing over lustre-MDT0000 [10717.508529] Lustre: server umount lustre-MDT0000 complete [10717.665855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.671286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.689249] Lustre: Skipped 2 previous similar messages [10729.011341] LustreError: 325190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.033912] LustreError: 325190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10738.160067] Lustre: 319066:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709252/real 1779709252] req@0000000096e1f8ad x1866160095194368/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779709268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10738.201809] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10738.459723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.392064] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1f13674f43f99815 [10748.406420] Lustre: MGC192.168.202.124@tcp: Connection restored to 0@lo (at 0@lo) [10748.426937] Lustre: Skipped 2 previous similar messages [10748.666026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.740608] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10751.507437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10754.031817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10754.109440] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10754.195799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10754.607392] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10762.926435] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10764.481732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10774.500257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10774.502861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10774.509449] Lustre: Skipped 1 previous similar message [10774.509859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10774.516164] Lustre: Skipped 3 previous similar messages [10777.773969] Lustre: server umount lustre-MDT0000 complete [10782.075902] LustreError: 325176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709312 with bad export cookie 2239247029873776661 [10782.093270] LustreError: 325176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10782.106482] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10782.354701] Lustre: server umount lustre-MDT0001 complete [10796.610382] Lustre: server umount lustre-OST0000 complete [10812.819574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10813.156753] LustreError: 330981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.173678] LustreError: 330981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [10813.240802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10817.351682] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10826.215965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.820609] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10836.972755] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10840.413104] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10847.398234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.952922] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10859.009353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10859.942760] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10867.169824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10867.178206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.190577] Lustre: Skipped 1 previous similar message [10867.196063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.208302] Lustre: Skipped 1 previous similar message [10871.165432] Lustre: server umount lustre-OST0000 complete [10877.920749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10877.924142] LustreError: Skipped 1 previous similar message [10877.944707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10877.956767] Lustre: Skipped 2 previous similar messages [10881.699504] Lustre: server umount lustre-MDT0000 complete [10885.391760] LustreError: 330962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709415 with bad export cookie 2239247029873777403 [10885.393401] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10885.421400] LustreError: 330962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10885.820685] Lustre: server umount lustre-MDT0001 complete [10897.174923] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [10899.701447] Key type lgssc unregistered [10900.058865] LNet: 334158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10900.070363] LNetError: 334158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10900.089463] LNet: Removed LNI 192.168.202.124@tcp [10901.006905] Key type .llcrypt unregistered [10901.009141] Key type ._llcrypt unregistered [10926.368719] Key type ._llcrypt registered [10926.371593] Key type .llcrypt registered [10926.474932] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10927.901484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10927.932205] alg: No test for adler32 (adler32-zlib) [10929.189758] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [10929.394222] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [10931.055247] Key type lgssc registered [10932.386061] Lustre: Echo OBD driver; http://www.lustre.org/ [10940.551059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10940.572806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.235153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.131565] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10952.673726] LustreError: 335657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.705024] LustreError: 335657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10957.087649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.475987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10963.592776] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10970.853979] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10975.632580] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10984.626324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.064944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10987.131379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10992.440712] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11000.005505] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11018.226333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11018.231627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11018.254850] Lustre: Skipped 1 previous similar message [11018.267512] Lustre: Skipped 2 previous similar messages [11022.531968] Lustre: server umount lustre-MDT0000 complete [11023.333461] LustreError: 336369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.354018] LustreError: 336369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11026.392667] LustreError: 335638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709556 with bad export cookie 4678048341727219288 [11026.403641] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.407411] LustreError: 335638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11026.726927] Lustre: server umount lustre-MDT0001 complete [11041.209300] Lustre: server umount lustre-OST0000 complete [11054.328617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.109448] LustreError: 339049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.217617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.193738] LustreError: 339050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.330280] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11065.315718] LustreError: 339049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.713124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.135877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11076.861708] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11083.499406] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11088.006371] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11096.674985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.102514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11100.232101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11103.960649] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11110.037777] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11123.172573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11123.175410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.212720] Lustre: Skipped 1 previous similar message [11123.226279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11125.730603] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.731343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11125.770955] Lustre: Skipped 2 previous similar messages [11128.953178] Lustre: server umount lustre-MDT0000 complete [11130.861676] LustreError: 339050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.902027] LustreError: 339050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11132.852523] LustreError: 339031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709662 with bad export cookie 4678048341727220401 [11132.858074] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11132.865789] LustreError: 339031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11133.435412] Lustre: server umount lustre-MDT0001 complete [11147.848549] Lustre: server umount lustre-OST0000 complete [11164.627434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.022192] LustreError: 342625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.095344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11169.387547] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11178.206379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11183.213558] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11189.369575] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11193.436263] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11201.336105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.057993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11202.070996] Lustre: Skipped 1 previous similar message [11210.240659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11211.652427] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11220.088423] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11226.138273] Lustre: server umount lustre-OST0000 complete [11234.784744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11234.796799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.826608] Lustre: Skipped 1 previous similar message [11234.831390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11240.932318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11240.936880] Lustre: Skipped 3 previous similar messages [11243.028668] Lustre: server umount lustre-MDT0000 complete [11244.960595] Lustre: 334970:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709758/real 1779709758] req@00000000f68df66b x1866160625774720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779709774 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11244.994832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.017438] Lustre: Skipped 1 previous similar message [11245.027377] LustreError: 342621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.049208] LustreError: 342621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11245.984107] Lustre: 334968:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709760/real 1779709760] req@0000000070400ffe x1866160625775872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779709776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11246.888424] LustreError: 342606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709776 with bad export cookie 4678048341727221423 [11246.901303] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11246.915384] LustreError: 342606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11247.302766] Lustre: server umount lustre-MDT0001 complete [11259.512493] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11262.677745] Key type lgssc unregistered [11262.919375] LNet: 345800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11262.927551] LNetError: 345800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11262.947798] LNet: Removed LNI 192.168.202.124@tcp [11263.840947] Key type .llcrypt unregistered [11263.844882] Key type ._llcrypt unregistered [11290.827191] Key type ._llcrypt registered [11290.837245] Key type .llcrypt registered [11290.973623] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11292.793933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11292.906146] alg: No test for adler32 (adler32-zlib) [11294.120350] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [11294.337734] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11296.031249] Key type lgssc registered [11297.656159] Lustre: Echo OBD driver; http://www.lustre.org/ [11307.036583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11307.083966] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11307.094561] LustreError: 347223:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11307.124789] LustreError: 347223:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11307.133479] LustreError: 347223:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11307.138993] LustreError: 347223:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11307.145681] LustreError: 347223:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11316.428136] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11316.439494] LustreError: 347621:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11316.461378] LustreError: 347621:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11316.469305] LustreError: 347621:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11316.478557] LustreError: 347621:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11316.495511] LustreError: 347621:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11340.930781] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11344.378467] Key type lgssc unregistered [11344.683537] LNet: 348608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11344.695615] LNetError: 348608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11344.711723] LNet: Removed LNI 192.168.202.124@tcp [11345.565725] Key type .llcrypt unregistered [11345.576657] Key type ._llcrypt unregistered [11366.614578] Key type ._llcrypt registered [11366.619220] Key type .llcrypt registered [11366.747615] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [11383.002530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11384.609954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11384.639124] alg: No test for adler32 (adler32-zlib) [11385.779551] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [11386.002910] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11387.655223] Key type lgssc registered [11388.985105] Lustre: Echo OBD driver; http://www.lustre.org/ [11430.814945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11430.837691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11432.192695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11432.244736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11432.362514] Lustre: lustre-MDT0000: new disk, initializing [11432.426248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11432.459389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11437.016418] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11448.400819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11448.513753] Lustre: 351732:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11448.558570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11448.560025] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11448.570833] Lustre: Skipped 1 previous similar message [11448.631695] Lustre: lustre-MDT0001: new disk, initializing [11448.721501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11448.759047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11448.774732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11453.650879] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11461.848402] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11466.304168] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11473.593779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11473.819712] Lustre: lustre-OST0000: new disk, initializing [11473.822818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11473.830238] Lustre: 353478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11473.921593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11481.167653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11481.177616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11481.263257] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11481.292823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11489.320539] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11494.073720] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11494.321840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11499.287922] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11499.585677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11502.049388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.059641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11502.076699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.170438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.173296] Lustre: Skipped 1 previous similar message [11508.115829] Lustre: server umount lustre-OST0000 complete [11512.291534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.316192] Lustre: Skipped 1 previous similar message [11512.324891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11512.329783] Lustre: Skipped 2 previous similar messages [11517.410578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11517.416044] Lustre: Skipped 1 previous similar message [11518.519583] Lustre: server umount lustre-MDT0000 complete [11522.522920] LustreError: 351724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710052 with bad export cookie 3887570056295169927 [11522.529496] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11522.542429] LustreError: 351724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11523.158154] Lustre: server umount lustre-MDT0001 complete [11532.619388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11534.330533] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:54:23 (1779710063) [11541.592784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.106554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11546.855532] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11547.115114] LustreError: 355790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.143300] LustreError: 355790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11552.226864] LustreError: 355789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.092063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.667173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11561.784375] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11568.651891] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11574.071620] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11584.346362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11584.794359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.477981] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11599.359682] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11605.575286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11605.751698] Lustre: lustre-OST0001: new disk, initializing [11605.754780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11605.759419] Lustre: 358286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11605.840429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11611.618948] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11613.777120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11613.793932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11613.906359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11618.932344] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11624.423903] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11624.426528] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11624.477476] Lustre: Skipped 1 previous similar message [11624.492270] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11628.912632] Lustre: server umount lustre-OST0001 complete [11629.546047] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.574531] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11634.671563] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11634.707315] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11639.813641] LustreError: 358731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.828429] LustreError: 358731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11649.507323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11649.523984] LustreError: Skipped 1 previous similar message [11649.531515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.543353] Lustre: Skipped 1 previous similar message [11649.552081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11649.558271] Lustre: Skipped 2 previous similar messages [11650.032356] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11650.064354] LustreError: 357386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11663.329055] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11663.569724] Lustre: server umount lustre-OST0000 complete [11669.472869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11669.482187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.497394] Lustre: Skipped 1 previous similar message [11674.838573] Lustre: server umount lustre-MDT0000 complete [11675.623764] LustreError: 355784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.650852] LustreError: 355784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11678.987984] LustreError: 355769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710209 with bad export cookie 3887570056295170809 [11678.994335] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11679.006802] LustreError: 355769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11685.530827] Lustre: server umount lustre-MDT0001 complete [11698.932916] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11702.474445] Key type lgssc unregistered [11702.961648] LNet: 360144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11702.966698] LNetError: 360144:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11702.979530] LNet: Removed LNI 192.168.202.124@tcp [11703.841970] Key type .llcrypt unregistered [11703.844470] Key type ._llcrypt unregistered [11760.690424] Key type ._llcrypt registered [11760.692770] Key type .llcrypt registered [11760.878968] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11762.241078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11762.360656] alg: No test for adler32 (adler32-zlib) [11763.592397] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [11763.845665] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11765.536953] Key type lgssc registered [11767.644393] Lustre: Echo OBD driver; http://www.lustre.org/ [11777.540606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11777.563760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.259747] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11779.304552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11779.736229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11784.983865] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11793.261051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11793.366843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11793.386528] Lustre: 362930:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11793.574726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11798.532598] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11805.009453] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11809.332965] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11813.859149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.863053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.873938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11815.912985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11815.913780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11818.692517] Lustre: server umount lustre-MDT0000 complete [11824.376438] LustreError: 362921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710354 with bad export cookie 7523200513874901038 [11824.377578] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11824.382470] LustreError: 362921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11824.708749] Lustre: server umount lustre-MDT0001 complete [11837.552759] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:59:25 (1779710365) [11845.527990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11845.972954] LustreError: 365218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.988935] LustreError: 365218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11846.055249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11851.237089] LustreError: 365219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.245031] LustreError: 365219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11851.632332] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11856.364859] LustreError: 365218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11861.492505] LustreError: 365219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.142512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11863.781105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.196941] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11877.029476] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11881.825380] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11891.184606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11891.511104] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11891.762739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11898.516104] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11903.999311] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11913.416931] Lustre: Permanently deactivating lustre-OST0000 [11913.465511] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11920.434584] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11922.118286] LustreError: 367816:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11922.142503] LustreError: 367816:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11922.151538] LustreError: 367816:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11925.989737] LustreError: 367865:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11925.996650] LustreError: 367865:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11931.411367] Lustre: server umount lustre-OST0000 complete [11934.502282] LustreError: 368126:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11934.515288] LustreError: 368126:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11940.832701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11940.835572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11940.846223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.846235] Lustre: Skipped 1 previous similar message [11940.885718] Lustre: Skipped 1 previous similar message [11945.953647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.960952] Lustre: Skipped 2 previous similar messages [11947.106183] Lustre: server umount lustre-MDT0000 complete [11950.904204] LustreError: 365198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710480 with bad export cookie 7523200513874901619 [11950.906463] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11950.934143] LustreError: 365198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11960.534673] Lustre: server umount lustre-MDT0001 complete [11969.745530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.043745] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11977.045412] LustreError: 369268:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11983.945715] Lustre: 369607:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.124@tcp [11986.564629] Lustre: server umount MGS complete [11999.170313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11999.769253] LustreError: 370332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.806340] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11999.824254] Lustre: Skipped 1 previous similar message [11999.910462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12005.859380] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12010.464927] LustreError: 370333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.497435] LustreError: 370333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12014.884395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12015.120343] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12019.791955] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12026.920588] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12031.855851] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12034.810825] Lustre: Permanently reactivating lustre-OST0000 [12034.838382] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12034.847509] Lustre: Skipped 2 previous similar messages [12062.175183] LustreError: 371838:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12069.618915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.917664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.924583] Lustre: Skipped 1 previous similar message [12076.549937] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12082.118856] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12087.272074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.313038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12093.658728] Lustre: server umount lustre-OST0000 complete [12102.113934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12102.146275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.166905] Lustre: Skipped 1 previous similar message [12102.191873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.200063] Lustre: Skipped 3 previous similar messages [12103.937550] Lustre: server umount lustre-MDT0000 complete [12107.473834] LustreError: 371680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710637 with bad export cookie 7523200513874902669 [12107.484803] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.491109] LustreError: 371680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12107.751174] Lustre: server umount lustre-MDT0001 complete [12118.176468] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12121.326894] Key type lgssc unregistered [12121.702495] LNet: 374292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12121.722729] LNetError: 374292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12121.765579] LNet: Removed LNI 192.168.202.124@tcp [12122.697092] Key type .llcrypt unregistered [12122.698772] Key type ._llcrypt unregistered [12141.636920] Key type ._llcrypt registered [12141.638627] Key type .llcrypt registered [12141.725925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12159.467872] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12161.138609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12161.334078] alg: No test for adler32 (adler32-zlib) [12162.602560] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12162.988448] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12164.807264] Key type lgssc registered [12166.164445] Lustre: Echo OBD driver; http://www.lustre.org/ [12204.978402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12205.015549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.408911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12206.461704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12206.618506] Lustre: lustre-MDT0000: new disk, initializing [12206.778804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12206.807346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12211.625765] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12222.841787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.983719] Lustre: 377416:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12223.048612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12223.053337] Lustre: Skipped 1 previous similar message [12223.127399] Lustre: lustre-MDT0001: new disk, initializing [12223.243935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.273615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12223.280759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12228.403931] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12238.542784] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12242.884974] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12251.753065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.147451] Lustre: lustre-OST0000: new disk, initializing [12252.150287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12252.165165] Lustre: 379157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12252.302174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12259.160279] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12260.919073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12260.930970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12261.136830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12267.705257] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12272.731976] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12272.949723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12278.431991] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12278.663525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12281.833615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12281.834392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12281.839223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12281.848790] Lustre: Skipped 1 previous similar message [12281.879848] LustreError: Skipped 1 previous similar message [12286.959595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12287.002204] Lustre: Skipped 2 previous similar messages [12292.068493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12292.083302] Lustre: Skipped 1 previous similar message [12295.135660] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12295.354294] Lustre: server umount lustre-OST0000 complete [12300.266889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12300.283131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12300.309795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12302.307403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.805995] Lustre: server umount lustre-MDT0000 complete [12309.562596] LustreError: 377406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710839 with bad export cookie 3936256522030088463 [12309.563430] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12309.568772] LustreError: 377406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12310.077363] Lustre: server umount lustre-MDT0001 complete [12318.733779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12320.217915] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:07:29 (1779710849) [12327.721515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.152875] LustreError: 381470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.164124] LustreError: 381470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12328.218579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12332.803925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12333.536589] LustreError: 381471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.680108] LustreError: 381470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.592711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.920141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12348.667928] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12354.776306] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12358.757309] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12366.117483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12366.613247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12372.523726] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12379.099342] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12381.474014] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12384.243178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12389.345134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.349526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.359332] Lustre: Skipped 1 previous similar message [12389.368812] Lustre: Skipped 2 previous similar messages [12395.535966] Lustre: server umount lustre-OST0000 complete [12399.585786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.608430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.615918] Lustre: Skipped 3 previous similar messages [12405.889049] Lustre: server umount lustre-MDT0000 complete [12409.778269] LustreError: 381450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710939 with bad export cookie 3936256522030089338 [12409.786057] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12409.787385] LustreError: 381450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12410.181797] Lustre: server umount lustre-MDT0001 complete [12421.240926] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12424.476217] Key type lgssc unregistered [12424.804110] LNet: 384840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12424.811755] LNetError: 384840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12425.829518] LNet: Removed LNI 192.168.202.124@tcp [12426.664532] Key type .llcrypt unregistered [12426.666876] Key type ._llcrypt unregistered [12449.269876] Key type ._llcrypt registered [12449.273032] Key type .llcrypt registered [12449.378630] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12449.661557] Key type .llcrypt unregistered [12449.665872] Key type ._llcrypt unregistered [12466.360381] Key type ._llcrypt registered [12466.362712] Key type .llcrypt registered [12466.454725] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12467.669632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12467.855087] alg: No test for adler32 (adler32-zlib) [12468.988113] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12469.334896] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12471.039377] Key type lgssc registered [12472.250202] Lustre: Echo OBD driver; http://www.lustre.org/ [12479.830949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12479.859761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12481.440114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12486.445665] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12491.745846] LustreError: 387033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.758928] LustreError: 387033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12495.080882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12495.406077] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.440869] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12495.449361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12500.450987] LustreError: 387033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.475301] LustreError: 387033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12501.619636] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12505.568943] LustreError: 387029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12505.601918] LustreError: 387029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12506.614348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12507.110295] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12511.722738] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12511.753913] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12513.810105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12514.138164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12518.601409] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12532.197480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12532.219670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12532.242592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12534.753209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12534.760211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12534.764056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12534.797593] Lustre: Skipped 1 previous similar message [12536.901127] Lustre: server umount lustre-OST0000 complete [12544.994742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.997699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12545.013693] Lustre: Skipped 1 previous similar message [12545.031395] Lustre: Skipped 1 previous similar message [12548.107315] Lustre: server umount lustre-MDT0000 complete [12550.115653] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.143898] LustreError: 387034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12552.966299] LustreError: 387014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711082 with bad export cookie 15396454472489318942 [12552.989674] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12553.418765] Lustre: server umount lustre-MDT0001 complete [12567.961983] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12571.616706] Key type lgssc unregistered [12572.032900] LNet: 390018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12572.041870] LNetError: 390018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12572.074224] LNet: Removed LNI 192.168.202.124@tcp [12573.256511] Key type .llcrypt unregistered [12573.259338] Key type ._llcrypt unregistered [12602.542594] Key type ._llcrypt registered [12602.550227] Key type .llcrypt registered [12602.751436] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12604.572376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12604.611040] alg: No test for adler32 (adler32-zlib) [12605.671465] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12605.935430] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12607.655207] Key type lgssc registered [12608.864739] Lustre: Echo OBD driver; http://www.lustre.org/ [12619.962074] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12619.981672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12635.689063] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12640.799638] LustreError: 391393:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [12667.551240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12674.750429] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12681.290341] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12689.429780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.961517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.050336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12695.014116] LustreError: 391418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.030246] LustreError: 391418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12695.410438] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12700.148307] LustreError: 391419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12702.178309] LustreError: 392317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12702.203158] LustreError: 392317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12705.691800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12706.086333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12711.976653] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12719.249482] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12724.545203] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12731.879024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12731.885482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.893644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12731.910206] Lustre: Skipped 1 previous similar message [12736.992784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12737.008954] Lustre: Skipped 1 previous similar message [12737.326313] Lustre: server umount lustre-OST0000 complete [12744.631177] Lustre: server umount lustre-MDT0000 complete [12745.187734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.216868] LustreError: 392316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12745.262320] LustreError: 392316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12749.222184] LustreError: 391400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711279 with bad export cookie 17712634854214315840 [12749.233166] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12749.245616] LustreError: 391400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12749.972938] Lustre: server umount lustre-MDT0001 complete [12762.955510] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12766.447211] Key type lgssc unregistered [12766.830628] LNet: 394607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12766.834742] LNetError: 394607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12767.914133] LNet: Removed LNI 192.168.202.124@tcp [12768.822388] Key type .llcrypt unregistered [12768.824547] Key type ._llcrypt unregistered [12795.454821] Key type ._llcrypt registered [12795.459623] Key type .llcrypt registered [12795.688378] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12797.807200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12797.831940] alg: No test for adler32 (adler32-zlib) [12799.082362] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12799.342971] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12801.007343] Key type lgssc registered [12802.455870] Lustre: Echo OBD driver; http://www.lustre.org/ [12810.828846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12810.851744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12812.551208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12817.020261] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12821.984878] LustreError: 396010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.015713] LustreError: 396010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12826.708036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12827.105922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12827.358340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12832.810450] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12839.740155] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12844.451253] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12853.834970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12854.452809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12857.537816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12864.091846] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12872.310278] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12878.310041] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.321948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12878.339851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12878.382291] Lustre: Skipped 2 previous similar messages [12879.414258] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [12883.070056] Lustre: server umount lustre-MDT0000 complete [12883.439258] LustreError: 396005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.482101] LustreError: 396005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12884.500116] LustreError: 397902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.782977] LustreError: 395990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.24@tcp arrived at 1779711416 with bad export cookie 16159023602582537352 [12888.546548] LustreError: 396010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.579601] LustreError: 396010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12889.041532] LustreError: 395991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711419 with bad export cookie 16159023602582536463 [12889.049943] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12889.070588] LustreError: 395991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12893.665233] LustreError: 397902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.665912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12893.690867] LustreError: 397902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12893.714577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12893.741585] Lustre: Skipped 2 previous similar messages [12893.749886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12895.587747] Lustre: server umount lustre-OST0000 complete [12904.531772] Lustre: server umount lustre-MDT0001 complete [12919.829434] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12923.051686] Key type lgssc unregistered [12923.455544] LNet: 399341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12923.460074] LNetError: 399341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12924.519465] LNet: Removed LNI 192.168.202.124@tcp [12925.743045] Key type .llcrypt unregistered [12925.747322] Key type ._llcrypt unregistered [12950.617699] Key type ._llcrypt registered [12950.619904] Key type .llcrypt registered [12950.723130] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12952.184034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12952.201399] alg: No test for adler32 (adler32-zlib) [12953.272539] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [12953.492642] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12955.143331] Key type lgssc registered [12956.414878] Lustre: Echo OBD driver; http://www.lustre.org/ [12964.102131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12964.111660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.632669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12970.684526] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12975.079288] LustreError: 400744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.108883] LustreError: 400744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12980.195422] LustreError: 400743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.427963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12980.971690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12985.405443] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12990.904782] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12994.710243] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13003.251902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13003.699954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13010.481429] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13010.887986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13016.350537] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13021.156630] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.186037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13021.200045] Lustre: Skipped 1 previous similar message [13024.312610] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [13026.292363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13026.303255] Lustre: Skipped 1 previous similar message [13027.548942] Lustre: server umount lustre-MDT0001 complete [13029.393983] LustreError: 400740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.397733] LustreError: 400740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.418523] LustreError: 400740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13036.513198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.520867] LustreError: 400744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.530021] Lustre: Skipped 1 previous similar message [13036.549574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13039.828918] Lustre: server umount lustre-OST0000 complete [13044.149643] Lustre: server umount lustre-MDT0000 complete [13058.060141] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13061.203425] Key type lgssc unregistered [13061.634627] LNet: 404073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13061.647283] LNetError: 404073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13061.662949] LNet: Removed LNI 192.168.202.124@tcp [13062.598685] Key type .llcrypt unregistered [13062.600330] Key type ._llcrypt unregistered [13090.493148] Key type ._llcrypt registered [13090.497288] Key type .llcrypt registered [13090.606760] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [13106.321171] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13108.496975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13108.550047] alg: No test for adler32 (adler32-zlib) [13109.901204] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [13110.098942] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13111.719188] Key type lgssc registered [13113.366137] Lustre: Echo OBD driver; http://www.lustre.org/ [13163.359337] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13183.140239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13183.200888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13184.766811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13184.843616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13185.013084] Lustre: lustre-MDT0000: new disk, initializing [13185.198551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13185.236592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13190.332077] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13211.664300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.053850] Lustre: lustre-OST0000: new disk, initializing [13212.061396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13212.071608] Lustre: Skipped 1 previous similar message [13212.083948] Lustre: 409043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13212.176827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13219.540822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13219.564677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13219.608670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13219.780255] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13230.643666] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13250.352702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13250.613848] Lustre: 408041:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13250.756522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13250.850238] Lustre: lustre-MDT0001: new disk, initializing [13251.052178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13251.168461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13251.190944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13257.325824] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13268.793299] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13269.049334] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13275.351498] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13275.622326] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13281.760683] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13281.763087] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13281.780671] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13281.795635] Lustre: Skipped 1 previous similar message [13281.811958] Lustre: Skipped 1 previous similar message [13285.932690] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [13288.065956] Lustre: server umount lustre-MDT0001 complete [13297.128455] LustreError: 408048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13297.151743] LustreError: 408048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13302.241512] LustreError: 408052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.266788] LustreError: 408052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13307.360822] LustreError: 408049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.377298] LustreError: 408049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13311.982980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13311.995660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13312.025711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13312.040555] Lustre: Skipped 2 previous similar messages [13312.488803] LustreError: 408053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.608388] LustreError: 408052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.609269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13317.646497] Lustre: Skipped 1 previous similar message [13322.728282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13323.231782] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13323.475200] Lustre: server umount lustre-OST0000 complete [13327.844386] LustreError: 408052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13327.891643] LustreError: 408052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13334.807021] Lustre: server umount lustre-MDT0000 complete [13352.313489] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13356.681572] Key type lgssc unregistered [13357.189072] LNet: 412614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13357.196637] LNetError: 412614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13357.220293] LNet: Removed LNI 192.168.202.124@tcp [13358.559096] Key type .llcrypt unregistered [13358.561069] Key type ._llcrypt unregistered [13415.161866] Key type ._llcrypt registered [13415.164685] Key type .llcrypt registered [13415.424691] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13415.932744] Key type .llcrypt unregistered [13415.934662] Key type ._llcrypt unregistered [13435.958360] Key type ._llcrypt registered [13435.961456] Key type .llcrypt registered [13436.154481] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13438.208480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13438.297746] alg: No test for adler32 (adler32-zlib) [13439.513696] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [13439.899903] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13441.695206] Key type lgssc registered [13443.091345] Lustre: Echo OBD driver; http://www.lustre.org/ [13476.138401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13476.171769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13477.455833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13477.488894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13477.567584] Lustre: lustre-MDT0000: new disk, initializing [13477.640621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13477.656700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13482.612207] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13497.297617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13497.475065] Lustre: 415963:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13497.534447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13497.536691] Lustre: Skipped 1 previous similar message [13497.695595] Lustre: lustre-MDT0001: new disk, initializing [13497.830991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13497.929088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13497.947053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13503.391559] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13513.890795] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13518.486817] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13529.521467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13530.125203] Lustre: lustre-OST0000: new disk, initializing [13530.133257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13530.147979] Lustre: 417706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13530.299519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13535.351189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13535.380851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13535.557294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13538.503746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13548.103237] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13556.195160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13556.203111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13556.216862] Lustre: Skipped 1 previous similar message [13561.313604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13561.320066] Lustre: Skipped 4 previous similar messages [13562.003295] Lustre: server umount lustre-MDT0000 complete [13567.296870] LustreError: 415955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712097 with bad export cookie 1138422919731733802 [13567.299207] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13567.311586] LustreError: 415955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13567.721386] Lustre: server umount lustre-MDT0001 complete [13582.413215] Lustre: server umount lustre-OST0000 complete [13597.702765] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:28:45 (1779712125) [13609.291475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13610.088144] LustreError: 419623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.103136] LustreError: 419623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13610.175963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13615.592129] LustreError: 419624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.720902] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13620.709248] LustreError: 419623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13625.834132] LustreError: 419624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13628.280794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13628.764863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13634.128244] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13640.993124] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13645.911288] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13667.239291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13667.518305] Lustre: Found index 0 for lustre-OST0000, updating log [13667.707505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13674.943863] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13681.092700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13681.955928] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13689.315784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13689.330556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13689.350986] Lustre: Skipped 1 previous similar message [13689.365552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13693.641757] Lustre: server umount lustre-OST0000 complete [13710.550534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13710.554305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.568666] Lustre: Skipped 2 previous similar messages [13710.601083] Lustre: Skipped 4 previous similar messages [13715.936886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13715.948933] Lustre: Skipped 1 previous similar message [13716.963987] LustreError: 419624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.989791] LustreError: 419624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13717.099774] Lustre: server umount lustre-MDT0000 complete [13721.938708] LustreError: 419605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712251 with bad export cookie 1138422919731735195 [13721.944929] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13721.955783] LustreError: 419605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13722.536246] Lustre: server umount lustre-MDT0001 complete [13728.799312] LNet: 1 local NIs in recovery (showing 1): 192.168.202.124@tcp [13733.726743] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:31:02 (1779712262) [14366.216959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14366.649579] LustreError: 425695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14366.676030] LustreError: 425695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14366.758843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14371.815972] LustreError: 425696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14372.104163] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14376.935707] LustreError: 425695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14381.624252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14382.035570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14387.161499] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14393.918469] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14398.330839] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14419.699769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14419.981571] Lustre: Found index 0 for lustre-OST0000, updating log [14420.005377] LNet: 1 local NIs in recovery (showing 1): 192.168.202.124@tcp [14420.285676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14426.462323] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14433.174729] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14439.905398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14439.913089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14439.934835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14450.144106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14450.465075] Lustre: server umount lustre-OST0000 complete [14456.081434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14456.088633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14456.103829] Lustre: Skipped 1 previous similar message [14456.119548] Lustre: Skipped 1 previous similar message [14458.148845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14458.155818] Lustre: Skipped 2 previous similar messages [14462.305566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14462.308236] Lustre: Skipped 8 previous similar messages [14462.445593] Lustre: server umount lustre-MDT0000 complete [14463.330875] LustreError: 425696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14463.361113] LustreError: 425696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14467.315788] LustreError: 425677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712997 with bad export cookie 1138422919731736280 [14467.330499] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14467.332479] LustreError: 425677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14467.845612] Lustre: server umount lustre-MDT0001 complete [14480.415297] LNet: 1 local NIs in recovery (showing 1): 192.168.202.124@tcp [14498.517577] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [14509.541133] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14547.408534] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:44:36 (1779713076) [14566.357185] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14569.649342] Key type lgssc unregistered [14570.132920] LNet: 431920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14570.142597] LNetError: 431920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14681.772907] LNet: Removed LNI 192.168.202.124@tcp [14682.703476] Key type .llcrypt unregistered [14682.705238] Key type ._llcrypt unregistered [14703.574550] Key type ._llcrypt registered [14703.576812] Key type .llcrypt registered [14703.673708] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14705.476619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14705.512292] alg: No test for adler32 (adler32-zlib) [14706.618948] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [14706.937520] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [14708.664411] Key type lgssc registered [14710.044658] Lustre: Echo OBD driver; http://www.lustre.org/ [14715.191722] LNet: Added LNI 192.168.202.124@tcp100 [8/256/0/180] [14716.294586] LNet: Added LNI 192.168.202.124@tcp101 [8/256/0/180] [14717.218972] LNet: Added LNI 192.168.202.124@tcp102 [8/256/0/180] [14718.379769] LNet: Added LNI 192.168.202.124@tcp103 [8/256/0/180] [14719.338984] LNet: Added LNI 192.168.202.124@tcp104 [8/256/0/180] [14720.281633] LNet: Added LNI 192.168.202.124@tcp105 [8/256/0/180] [14721.538293] LNet: Added LNI 192.168.202.124@tcp106 [8/256/0/180] [14722.557733] LNet: Added LNI 192.168.202.124@tcp107 [8/256/0/180] [14723.472430] LNet: Added LNI 192.168.202.124@tcp108 [8/256/0/180] [14724.486993] LNet: Added LNI 192.168.202.124@tcp109 [8/256/0/180] [14725.586281] LNet: Added LNI 192.168.202.124@tcp110 [8/256/0/180] [14726.434649] LNet: Added LNI 192.168.202.124@tcp111 [8/256/0/180] [14727.492924] LNet: Added LNI 192.168.202.124@tcp112 [8/256/0/180] [14728.559732] LNet: Added LNI 192.168.202.124@tcp113 [8/256/0/180] [14729.719163] LNet: Added LNI 192.168.202.124@tcp114 [8/256/0/180] [14730.774129] LNet: Added LNI 192.168.202.124@tcp115 [8/256/0/180] [14731.773927] LNet: Added LNI 192.168.202.124@tcp116 [8/256/0/180] [14732.607039] LNet: Added LNI 192.168.202.124@tcp117 [8/256/0/180] [14733.675624] LNet: Added LNI 192.168.202.124@tcp118 [8/256/0/180] [14734.654932] LNet: Added LNI 192.168.202.124@tcp119 [8/256/0/180] [14735.675562] LNet: Added LNI 192.168.202.124@tcp120 [8/256/0/180] [14737.130407] LNet: Added LNI 192.168.202.124@tcp121 [8/256/0/180] [14738.123679] LNet: Added LNI 192.168.202.124@tcp122 [8/256/0/180] [14738.996505] LNet: Added LNI 192.168.202.124@tcp123 [8/256/0/180] [14740.293771] LNet: Added LNI 192.168.202.124@tcp124 [8/256/0/180] [14741.738824] LNet: Added LNI 192.168.202.124@tcp125 [8/256/0/180] [14742.748765] LNet: Added LNI 192.168.202.124@tcp126 [8/256/0/180] [14743.617556] LNet: Added LNI 192.168.202.124@tcp127 [8/256/0/180] [14744.687622] LNet: Added LNI 192.168.202.124@tcp128 [8/256/0/180] [14745.717988] LNet: Added LNI 192.168.202.124@tcp129 [8/256/0/180] [14746.581602] LNet: Added LNI 192.168.202.124@tcp130 [8/256/0/180] [14747.490118] LNet: Added LNI 192.168.202.124@tcp131 [8/256/0/180] [14748.389608] LNet: Added LNI 192.168.202.124@tcp132 [8/256/0/180] [14749.306189] LNet: Added LNI 192.168.202.124@tcp133 [8/256/0/180] [14750.193504] LNet: Added LNI 192.168.202.124@tcp134 [8/256/0/180] [14750.997140] LNet: Added LNI 192.168.202.124@tcp135 [8/256/0/180] [14759.159865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14759.205441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14760.398969] Lustre: 434935:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp130: rc = -75 [14760.544642] LNetError: 434958:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14760.595114] LNetError: Protocol error connecting to 192.168.202.124@tcp115 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14760.616513] LNetError: Skipped 2 previous similar messages [14760.811681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14760.867252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14760.964596] Lustre: lustre-MDT0000: new disk, initializing [14761.057874] Lustre: 434991:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp130: rc = -75 [14761.081817] Lustre: 434991:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14761.115404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14761.154213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14761.249693] LNet: 2 local NIs in recovery (showing 2): 192.168.202.124@tcp128, 192.168.202.124@tcp129 [14761.253378] LNetError: 434962:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14761.280933] LNetError: 434962:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14761.290088] LNetError: Protocol error connecting to 192.168.202.124@tcp104 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14761.303231] LNetError: Skipped 1 previous similar message [14766.529566] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14780.944247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14781.122186] Lustre: 434951:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14781.238715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14781.252260] Lustre: Skipped 1 previous similar message [14781.414311] Lustre: lustre-MDT0001: new disk, initializing [14781.478208] Lustre: 435738:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp130: rc = -75 [14781.501080] Lustre: 435738:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14781.806115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14781.899425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14781.924089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14788.365936] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14790.944931] LNetError: 432393:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14790.979487] LNetError: 432393:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14790.994041] LNetError: Protocol error connecting to 192.168.202.124@tcp127 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14791.241971] Lustre: 436086:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp130: rc = -75 [14791.247627] Lustre: 436086:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14794.016618] LNetError: 432395:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14794.037729] LNetError: Protocol error connecting to 192.168.202.124@tcp104 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14798.116147] LNetError: 434963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14798.125388] LNetError: 434963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14798.134569] LNetError: Protocol error connecting to 192.168.202.124@tcp126 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14798.142597] LNetError: Skipped 2 previous similar messages [14798.451969] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14802.969676] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14806.328913] LNetError: 432392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14806.342722] LNetError: 432392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14806.373663] LNetError: Protocol error connecting to 192.168.202.124@tcp104 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14806.409513] LNetError: Skipped 2 previous similar messages [14811.065944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14811.427547] Lustre: lustre-OST0000: new disk, initializing [14811.433952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14811.449497] Lustre: 436704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14811.597923] Lustre: 436709:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp130: rc = -75 [14811.612054] Lustre: 436709:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14811.723722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14819.485659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14819.505957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14819.599479] Lustre: 437010:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.124@tcp: can't add NID 192.168.202.124@tcp132: rc = -75 [14819.605923] Lustre: 437010:0:(lustre_peer.c:105:class_add_uuid()) Skipped 19 previous similar messages [14819.753675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14819.971541] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [14822.690462] LNetError: 432394:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14822.707881] LNetError: 432394:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14822.718872] LNetError: Protocol error connecting to 192.168.202.124@tcp128 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14822.741674] LNetError: Skipped 7 previous similar messages [14829.643586] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14840.290964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14840.304255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14840.318764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14843.607085] Lustre: server umount lustre-OST0000 complete [14850.531630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14850.537247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14850.546778] Lustre: Skipped 2 previous similar messages [14850.567355] Lustre: Skipped 2 previous similar messages [14855.150239] Lustre: server umount lustre-MDT0000 complete [14859.483718] LustreError: 434945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713389 with bad export cookie 4782487141317543386 [14859.488674] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14859.499106] LustreError: 434945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14859.552507] LNetError: 432394:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.124 [14859.564871] LNetError: 432394:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14859.569959] LNetError: Protocol error connecting to 192.168.202.124@tcp115 at host 192.168.202.124:988: is it running a compatible version of Lustre? [14859.576911] LNetError: Skipped 9 previous similar messages [14859.822607] Lustre: server umount lustre-MDT0001 complete [14874.374742] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [14877.897814] Key type lgssc unregistered [14878.347835] LNet: 438487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14878.360273] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14879.391191] LNet: Removed LNI 192.168.202.124@tcp [14879.397594] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14879.409884] LNet: Removed LNI 192.168.202.124@tcp100 [14880.479167] LNet: Removed LNI 192.168.202.124@tcp101 [14880.489570] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14880.495149] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14880.513238] LNet: Removed LNI 192.168.202.124@tcp102 [14881.567253] LNet: Removed LNI 192.168.202.124@tcp103 [14882.591228] LNet: Removed LNI 192.168.202.124@tcp104 [14882.595257] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14882.607228] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14882.618749] LNet: Removed LNI 192.168.202.124@tcp105 [14882.625779] LNet: Removed LNI 192.168.202.124@tcp106 [14883.679180] LNet: Removed LNI 192.168.202.124@tcp107 [14884.703296] LNet: Removed LNI 192.168.202.124@tcp108 [14885.731667] LNet: Removed LNI 192.168.202.124@tcp109 [14886.751490] LNet: Removed LNI 192.168.202.124@tcp110 [14886.759662] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14886.768459] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [14886.776933] LNet: Removed LNI 192.168.202.124@tcp111 [14887.839708] LNet: Removed LNI 192.168.202.124@tcp112 [14888.863297] LNet: Removed LNI 192.168.202.124@tcp113 [14889.887338] LNet: Removed LNI 192.168.202.124@tcp114 [14889.893570] LNet: Removed LNI 192.168.202.124@tcp115 [14890.911341] LNet: Removed LNI 192.168.202.124@tcp116 [14891.935995] LNet: Removed LNI 192.168.202.124@tcp117 [14892.960215] LNet: Removed LNI 192.168.202.124@tcp118 [14893.983222] LNet: Removed LNI 192.168.202.124@tcp119 [14895.007333] LNet: Removed LNI 192.168.202.124@tcp120 [14895.013258] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14895.025315] LNetError: 438487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [14895.030870] LNet: Removed LNI 192.168.202.124@tcp121 [14896.097583] LNet: Removed LNI 192.168.202.124@tcp122 [14897.119360] LNet: Removed LNI 192.168.202.124@tcp123 [14898.143384] LNet: Removed LNI 192.168.202.124@tcp124 [14899.167325] LNet: Removed LNI 192.168.202.124@tcp125 [14900.191318] LNet: Removed LNI 192.168.202.124@tcp126 [14901.215388] LNet: Removed LNI 192.168.202.124@tcp127 [14902.239182] LNet: Removed LNI 192.168.202.124@tcp128 [14903.263208] LNet: Removed LNI 192.168.202.124@tcp129 [14903.265306] LNet: Removed LNI 192.168.202.124@tcp130 [14903.267183] LNet: Removed LNI 192.168.202.124@tcp131 [14903.272487] LNet: Removed LNI 192.168.202.124@tcp132 [14903.277883] LNet: Removed LNI 192.168.202.124@tcp133 [14903.281180] LNet: Removed LNI 192.168.202.124@tcp134 [14903.291505] LNet: Removed LNI 192.168.202.124@tcp135 [14904.144931] Key type .llcrypt unregistered [14904.157538] Key type ._llcrypt unregistered [14930.625668] Key type ._llcrypt registered [14930.629739] Key type .llcrypt registered [14930.792683] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [14948.438472] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [14950.153774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14950.312091] alg: No test for adler32 (adler32-zlib) [14951.446272] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [14951.703339] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [14953.383386] Key type lgssc registered [14954.846333] Lustre: Echo OBD driver; http://www.lustre.org/ [14993.595661] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:52:02 (1779713522) [15011.576790] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [15014.480393] Key type lgssc unregistered [15014.816832] LNet: 442528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15014.826313] LNetError: 442528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15014.856316] LNet: Removed LNI 192.168.202.124@tcp [15015.732428] Key type .llcrypt unregistered [15015.735851] Key type ._llcrypt unregistered [15034.867295] Key type ._llcrypt registered [15034.869841] Key type .llcrypt registered [15034.971349] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [15036.170563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15036.198439] alg: No test for adler32 (adler32-zlib) [15037.420974] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [15037.814309] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [15039.519181] Key type lgssc registered [15041.023146] Lustre: Echo OBD driver; http://www.lustre.org/ [15053.371523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15053.402098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15054.648502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15054.671453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15054.749390] Lustre: lustre-MDT0000: new disk, initializing [15054.847576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15054.867706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15059.954070] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15073.231630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15073.477334] Lustre: 443727:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15073.592667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15073.601984] Lustre: Skipped 1 previous similar message [15073.748617] Lustre: lustre-MDT0001: new disk, initializing [15073.970599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15074.026438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15074.041675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15080.447925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15089.356618] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15093.913902] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15102.202455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15102.559702] Lustre: lustre-OST0000: new disk, initializing [15102.563623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15102.570450] Lustre: 445467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15102.684576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15109.249165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15109.269085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15109.394054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15110.245968] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15118.813462] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15122.494920] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing lnet_if_list [15130.027373] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15130.083950] LNet: Added LNI 192.168.202.124@tcp10 [8/256/0/180] [15138.174349] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15138.208677] LNet: Added LNI 192.168.202.124@tcp11 [8/256/0/180] [15142.290250] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15142.321152] LNet: Added LNI 192.168.202.124@tcp12 [8/256/0/180] [15147.910661] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15147.972349] LNet: Added LNI 192.168.202.124@tcp13 [8/256/0/180] [15153.062459] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15153.113780] LNet: Added LNI 192.168.202.124@tcp14 [8/256/0/180] [15157.865938] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15157.900883] LNet: Added LNI 192.168.202.124@tcp15 [8/256/0/180] [15164.201943] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15164.228759] LNet: Added LNI 192.168.202.124@tcp16 [8/256/0/180] [15170.880706] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15170.899786] LNet: Added LNI 192.168.202.124@tcp17 [8/256/0/180] [15177.028780] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15177.137845] LNet: Added LNI 192.168.202.124@tcp18 [8/256/0/180] [15183.690916] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15183.717752] LNet: Added LNI 192.168.202.124@tcp19 [8/256/0/180] [15189.107990] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15189.143921] LNet: Added LNI 192.168.202.124@tcp20 [8/256/0/180] [15194.715371] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15194.740428] LNet: Added LNI 192.168.202.124@tcp21 [8/256/0/180] [15199.728447] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15199.749721] LNet: Added LNI 192.168.202.124@tcp22 [8/256/0/180] [15205.108511] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15205.149367] LNet: Added LNI 192.168.202.124@tcp23 [8/256/0/180] [15212.900721] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15212.935890] LNet: Added LNI 192.168.202.124@tcp24 [8/256/0/180] [15221.227386] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15221.279269] LNet: Added LNI 192.168.202.124@tcp25 [8/256/0/180] [15229.178631] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15229.206773] LNet: Added LNI 192.168.202.124@tcp26 [8/256/0/180] [15235.974383] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15236.027609] LNet: Added LNI 192.168.202.124@tcp27 [8/256/0/180] [15243.842446] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15243.910373] LNet: Added LNI 192.168.202.124@tcp28 [8/256/0/180] [15250.211981] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15250.259196] LNet: Added LNI 192.168.202.124@tcp29 [8/256/0/180] [15257.057930] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15257.082369] LNet: Added LNI 192.168.202.124@tcp30 [8/256/0/180] [15264.465694] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15264.493629] LNet: Added LNI 192.168.202.124@tcp31 [8/256/0/180] [15270.859505] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15270.893548] LNet: Added LNI 192.168.202.124@tcp32 [8/256/0/180] [15278.025467] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15278.081367] LNet: Added LNI 192.168.202.124@tcp33 [8/256/0/180] [15285.093579] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15285.114563] LNet: Added LNI 192.168.202.124@tcp34 [8/256/0/180] [15293.038892] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15293.071197] LNet: Added LNI 192.168.202.124@tcp35 [8/256/0/180] [15301.489584] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15301.516356] LNet: Added LNI 192.168.202.124@tcp36 [8/256/0/180] [15309.270626] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15309.316411] LNet: Added LNI 192.168.202.124@tcp37 [8/256/0/180] [15316.803647] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15316.830811] LNet: Added LNI 192.168.202.124@tcp38 [8/256/0/180] [15324.700979] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15324.738059] LNet: Added LNI 192.168.202.124@tcp39 [8/256/0/180] [15332.555050] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15332.583984] LNet: Added LNI 192.168.202.124@tcp40 [8/256/0/180] [15340.079132] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15340.102348] LNet: Added LNI 192.168.202.124@tcp41 [8/256/0/180] [15348.876480] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15348.901402] LNet: Added LNI 192.168.202.124@tcp42 [8/256/0/180] [15356.871675] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15356.967636] LNet: Added LNI 192.168.202.124@tcp43 [8/256/0/180] [15364.446597] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15364.477422] LNet: Added LNI 192.168.202.124@tcp44 [8/256/0/180] [15373.584954] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15373.625887] LNet: Added LNI 192.168.202.124@tcp45 [8/256/0/180] [15382.025900] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15382.047220] LNet: Added LNI 192.168.202.124@tcp46 [8/256/0/180] [15390.445640] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15390.469407] LNet: Added LNI 192.168.202.124@tcp47 [8/256/0/180] [15398.786188] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15398.812697] LNet: Added LNI 192.168.202.124@tcp48 [8/256/0/180] [15407.994901] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15408.046704] LNet: Added LNI 192.168.202.124@tcp49 [8/256/0/180] [15419.404198] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15419.432453] LNet: Added LNI 192.168.202.124@tcp50 [8/256/0/180] [15440.356765] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15440.371250] Lustre: Skipped 2 previous similar messages [15440.376137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15440.386876] Lustre: Skipped 2 previous similar messages [15445.484051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15445.500982] Lustre: Skipped 2 previous similar messages [15450.593278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15450.611487] Lustre: Skipped 2 previous similar messages [15454.176692] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15454.572789] Lustre: server umount lustre-MDT0000 complete [15458.557799] LustreError: 443718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713988 with bad export cookie 13179425062173332926 [15458.562120] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15458.576581] LustreError: 443718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15459.144606] Lustre: server umount lustre-MDT0001 complete [15476.191346] Lustre: 442998:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779713990/real 1779713990] req@000000008b1a1d8a x1866164933419648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779714006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15476.221979] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15479.292443] Lustre: 442999:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779713993/real 1779713993] req@00000000c097dded x1866164933419776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779714009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15479.954154] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [15481.377030] Lustre: 443000:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779713995/real 1779713995] req@000000001359d4cf x1866164933419904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779714011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15484.383219] Lustre: 442998:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779713998/real 1779713998] req@00000000d06ce0ee x1866164933420160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779714014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15489.072174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15490.656087] Lustre: 442998:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779714004/real 1779714004] req@000000004fa78094 x1866164933420288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779714020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15503.905616] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb6e6b94436b8e092 [15503.930508] Lustre: MGC192.168.202.124@tcp: Connection restored to 0@lo (at 0@lo) [15504.402774] LustreError: 445461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15504.418984] LustreError: 445461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [15504.546757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15506.532251] LustreError: 445462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15509.651803] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15510.646365] LustreError: 445461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15510.656799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15516.140410] LustreError: 445462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15519.079154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15519.585798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15521.653415] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15525.222393] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15532.132618] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15539.179150] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15562.727381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15562.730524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15562.749190] Lustre: Skipped 1 previous similar message [15562.759709] Lustre: Skipped 1 previous similar message [15567.831664] Lustre: server umount lustre-OST0000 complete [15572.964700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15572.968700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15572.984975] Lustre: Skipped 1 previous similar message [15573.016595] Lustre: Skipped 1 previous similar message [15578.487907] Lustre: server umount lustre-MDT0000 complete [15582.182833] LustreError: 468876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15582.196369] LustreError: 468876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15582.997125] LustreError: 448275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779714113 with bad export cookie 13179425062173335698 [15583.008488] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15583.022387] LustreError: 448275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15583.617129] Lustre: server umount lustre-MDT0001 complete [15596.340331] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [15599.790767] Key type lgssc unregistered [15600.251868] LNet: 471442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15600.263991] LNetError: 471442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15600.267647] LNet: Removed LNI 192.168.202.124@tcp [15600.272198] LNet: Removed LNI 192.168.202.124@tcp10 [15600.274131] LNet: Removed LNI 192.168.202.124@tcp11 [15600.277977] LNet: Removed LNI 192.168.202.124@tcp12 [15600.282676] LNet: Removed LNI 192.168.202.124@tcp13 [15600.288867] LNet: Removed LNI 192.168.202.124@tcp14 [15600.293977] LNet: Removed LNI 192.168.202.124@tcp15 [15600.305333] LNet: Removed LNI 192.168.202.124@tcp16 [15600.307377] LNet: Removed LNI 192.168.202.124@tcp17 [15600.309081] LNet: Removed LNI 192.168.202.124@tcp18 [15600.310582] LNet: Removed LNI 192.168.202.124@tcp19 [15600.317520] LNet: Removed LNI 192.168.202.124@tcp20 [15600.323904] LNet: Removed LNI 192.168.202.124@tcp21 [15600.328704] LNet: Removed LNI 192.168.202.124@tcp22 [15600.334042] LNet: Removed LNI 192.168.202.124@tcp23 [15600.337201] LNet: Removed LNI 192.168.202.124@tcp24 [15600.340967] LNet: Removed LNI 192.168.202.124@tcp25 [15600.345198] LNet: Removed LNI 192.168.202.124@tcp26 [15600.347207] LNet: Removed LNI 192.168.202.124@tcp27 [15600.350160] LNet: Removed LNI 192.168.202.124@tcp28 [15600.362199] LNet: Removed LNI 192.168.202.124@tcp29 [15600.367385] LNet: Removed LNI 192.168.202.124@tcp30 [15600.370654] LNet: Removed LNI 192.168.202.124@tcp31 [15600.382400] LNet: Removed LNI 192.168.202.124@tcp32 [15600.387928] LNet: Removed LNI 192.168.202.124@tcp33 [15600.389847] LNet: Removed LNI 192.168.202.124@tcp34 [15600.396724] LNet: Removed LNI 192.168.202.124@tcp35 [15600.401704] LNet: Removed LNI 192.168.202.124@tcp36 [15600.406702] LNet: Removed LNI 192.168.202.124@tcp37 [15600.409858] LNet: Removed LNI 192.168.202.124@tcp38 [15600.414981] LNet: Removed LNI 192.168.202.124@tcp39 [15600.417720] LNet: Removed LNI 192.168.202.124@tcp40 [15600.425476] LNet: Removed LNI 192.168.202.124@tcp41 [15600.428218] LNet: Removed LNI 192.168.202.124@tcp42 [15600.432812] LNet: Removed LNI 192.168.202.124@tcp43 [15600.436929] LNet: Removed LNI 192.168.202.124@tcp44 [15600.438843] LNet: Removed LNI 192.168.202.124@tcp45 [15600.444850] LNet: Removed LNI 192.168.202.124@tcp46 [15600.458035] LNet: Removed LNI 192.168.202.124@tcp47 [15600.464115] LNet: Removed LNI 192.168.202.124@tcp48 [15600.467674] LNet: Removed LNI 192.168.202.124@tcp49 [15600.487278] LNet: Removed LNI 192.168.202.124@tcp50 [15601.265819] Key type .llcrypt unregistered [15601.269102] Key type ._llcrypt unregistered [15625.271459] Key type ._llcrypt registered [15625.274124] Key type .llcrypt registered [15625.445154] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [15640.668840] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [15642.673423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15643.351832] alg: No test for adler32 (adler32-zlib) [15644.629066] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [15644.937193] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [15646.623789] Key type lgssc registered [15648.637249] Lustre: Echo OBD driver; http://www.lustre.org/ [15693.406872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15693.442756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15694.862105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15695.013817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15695.267746] Lustre: lustre-MDT0000: new disk, initializing [15695.456701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15695.527846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15701.278723] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15712.930029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15713.141606] Lustre: 474608:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15713.238749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15713.244987] Lustre: Skipped 1 previous similar message [15713.369699] Lustre: lustre-MDT0001: new disk, initializing [15713.536203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15713.576823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15713.596573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15717.882636] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15727.925614] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15732.406092] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15740.885686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15741.203204] Lustre: *** cfs_fail_loc=90f, val=0*** [15741.305689] Lustre: lustre-OST0000: new disk, initializing [15741.309403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15741.312311] Lustre: 476400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15741.424794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15748.548972] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15748.682022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15748.695265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15748.776582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15758.126466] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15769.571512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15769.573457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15769.579353] LustreError: Skipped 1 previous similar message [15769.605229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15769.614261] Lustre: Skipped 1 previous similar message [15774.689444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15774.704595] Lustre: Skipped 1 previous similar message [15779.814396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15779.828561] Lustre: Skipped 1 previous similar message [15781.346733] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15781.550032] Lustre: server umount lustre-OST0000 complete [15790.050062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15790.055529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15790.069811] Lustre: Skipped 2 previous similar messages [15790.088247] Lustre: Skipped 1 previous similar message [15793.581243] Lustre: server umount lustre-MDT0000 complete [15797.855260] LustreError: 474599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779714327 with bad export cookie 12069769777152960593 [15797.862274] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15797.866476] LustreError: 474599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15798.325728] Lustre: server umount lustre-MDT0001 complete [15809.615803] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [15813.020511] Key type lgssc unregistered [15813.426702] LNet: 478178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15813.444623] LNetError: 478178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15813.469638] LNet: Removed LNI 192.168.202.124@tcp [15814.588183] Key type .llcrypt unregistered [15814.595883] Key type ._llcrypt unregistered [15837.295905] Key type ._llcrypt registered [15837.298638] Key type .llcrypt registered [15837.505401] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [15838.953927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15839.010103] alg: No test for adler32 (adler32-zlib) [15840.346517] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [15840.600334] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [15842.239579] Key type lgssc registered [15843.729419] Lustre: Echo OBD driver; http://www.lustre.org/ [15853.024293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15853.052575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15854.704524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15859.146589] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15864.291679] LustreError: 479432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15864.303254] LustreError: 479432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15869.412651] LustreError: 479433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15869.538804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15870.124093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15875.484451] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15881.873879] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15886.036510] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15893.889993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15894.221607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15900.758533] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15908.216452] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15919.657444] loop0: detected capacity change from 0 to 204800000 [15919.892586] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15919.901474] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15919.913142] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15927.247315] loop0: detected capacity change from 0 to 204800000 [15927.479180] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15927.499352] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15927.517185] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15934.248773] loop0: detected capacity change from 0 to 204800000 [15940.754925] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15940.960375] Lustre: fs15246-OST0000: new disk, initializing [15940.968629] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15940.992520] Lustre: 482686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15941.070656] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15948.099606] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15960.784934] loop1: detected capacity change from 0 to 204800000 [15967.275847] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15967.461219] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15967.543607] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15967.684936] Lustre: fs15246-MDT0000: new disk, initializing [15967.877808] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15967.944283] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15968.093710] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15969.127094] Lustre: 483795:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000135385f8 x1866165775287552/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779714515 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15973.140453] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [15974.269976] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15974.293233] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15974.404046] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15998.435063] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15998.473665] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16001.825848] Lustre: server umount fs15246-MDT0000 complete [16003.559164] LustreError: 480893:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16008.679189] LustreError: 479429:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16011.733910] Lustre: server umount fs15246-OST0000 complete [16027.618523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16027.627581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16027.646168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16028.642905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16028.659613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16028.674031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16030.763942] Lustre: server umount lustre-OST0000 complete [16038.880953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16038.889933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16038.918376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16038.921104] Lustre: Skipped 2 previous similar messages [16042.451789] Lustre: server umount lustre-MDT0000 complete [16044.513117] LustreError: 480144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16044.533642] LustreError: 480144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16046.598553] LustreError: 479413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779714576 with bad export cookie 3060361930719867296 [16046.601185] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16046.617662] LustreError: 479413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16046.920706] Lustre: server umount lustre-MDT0001 complete [16059.350792] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [16062.358149] Key type lgssc unregistered [16062.685639] LNet: 486858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16062.696547] LNetError: 486858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16063.719466] LNet: Removed LNI 192.168.202.124@tcp [16064.775571] Key type .llcrypt unregistered [16064.777749] Key type ._llcrypt unregistered [16085.716460] Key type ._llcrypt registered [16085.718281] Key type .llcrypt registered [16085.860751] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [16087.374606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16087.567586] alg: No test for adler32 (adler32-zlib) [16088.695613] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [16088.903351] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [16090.560617] Key type lgssc registered [16091.505679] Lustre: Echo OBD driver; http://www.lustre.org/ [16133.790893] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [16143.114239] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [16181.119413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16181.148981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16182.530734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16182.582988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16182.729718] Lustre: lustre-MDT0000: new disk, initializing [16182.828427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16182.845797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16187.182363] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [16197.981843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16198.180819] Lustre: 491277:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16198.284883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16198.289079] Lustre: Skipped 1 previous similar message [16198.344272] Lustre: lustre-MDT0001: new disk, initializing [16198.424105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16198.462073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16198.476209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16203.257828] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [16211.519390] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16215.583023] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16222.992752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16223.271969] Lustre: lustre-OST0000: new disk, initializing [16223.280465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16223.287700] Lustre: 493018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16223.403231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16230.803535] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [16233.568804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16233.580492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16233.741541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16238.809409] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16243.524722] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16243.854312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16250.349446] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16250.741667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16254.434025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16254.444846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16254.461392] LustreError: Skipped 1 previous similar message [16254.470687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16254.484248] Lustre: Skipped 1 previous similar message [16254.519459] Lustre: Skipped 1 previous similar message [16260.083568] Lustre: server umount lustre-OST0000 complete [16265.184351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16265.193338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16265.207317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16269.793369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16269.811856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16271.318402] Lustre: server umount lustre-MDT0000 complete [16274.912817] LustreError: 491288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16274.929415] LustreError: 491288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [16276.337763] LustreError: 491269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779714806 with bad export cookie 5850490865692769190 [16276.343053] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16277.026440] Lustre: server umount lustre-MDT0001 complete [16306.752941] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [16321.817706] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [16359.257149] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16032 sec ========= 09:14:47 (1779714887) [16361.293850] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:14:49 (1779714889) === [16365.576870] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:14:54 (1779714894) === [16411.227448] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [16414.859428] Key type lgssc unregistered [16415.101546] LNet: 499489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16415.122698] LNetError: 499489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16415.144796] LNet: Removed LNI 192.168.202.124@tcp [16416.124860] Key type .llcrypt unregistered [16416.133047] Key type ._llcrypt unregistered