[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 536339399 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002366] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007429] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.021811] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025878] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027158] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] *** VALIDATE tmpfs *** [ 0.030291] *** VALIDATE proc *** [ 0.031223] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036038] Spectre V2 : User space: Vulnerable [ 0.037013] Speculative Store Bypass: Vulnerable [ 0.040239] debug: unmapping init [mem 0xffffffffb2259000-0xffffffffb2260fff] [ 0.043196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045035] ... version: 2 [ 0.046020] ... bit width: 48 [ 0.047017] ... generic registers: 4 [ 0.048024] ... value mask: 0000ffffffffffff [ 0.049023] ... max period: 00007fffffffffff [ 0.050038] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052439] rcu: Hierarchical SRCU implementation. [ 0.054824] smp: Bringing up secondary CPUs ... [ 0.055713] x86: Booting SMP configuration: [ 0.056041] .... node #0, CPUs: #1 #2 #3 [ 0.064131] smp: Brought up 1 node, 4 CPUs [ 0.066021] smpboot: Max logical packages: 1 [ 0.067019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.201695] node 0 deferred pages initialised in 132ms [ 0.205237] devtmpfs: initialized [ 0.206249] x86/mm: Memory block size: 128MB [ 0.208449] gcov: version magic: 0x41383552 [ 0.210302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.211086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.212484] pinctrl core: initialized pinctrl subsystem [ 0.213381] [ 0.214017] ************************************************************* [ 0.215026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.216023] ** ** [ 0.217020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.218022] ** ** [ 0.219050] ** This means that this kernel is built to expose internal ** [ 0.220020] ** IOMMU data structures, which may compromise security on ** [ 0.221028] ** your system. ** [ 0.222026] ** ** [ 0.223026] ** If you see this message and you are not debugging the ** [ 0.224017] ** kernel, report this immediately to your vendor! ** [ 0.225027] ** ** [ 0.226024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227022] ************************************************************* [ 0.229108] NET: Registered protocol family 16 [ 0.230454] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.231087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.232114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234118] cpuidle: using governor menu [ 0.235000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.238580] PCI: Using configuration type 1 for base access [ 0.240217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253113] cryptd: max_cpu_qlen set to 1000 [ 0.255332] ACPI: Added _OSI(Module Device) [ 0.257019] ACPI: Added _OSI(Processor Device) [ 0.258018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.260018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.271743] ACPI: Interpreter enabled [ 0.273076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275013] ACPI: Using IOAPIC for interrupt routing [ 0.277802] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.293000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.296046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308419] acpiphp: Slot [2] registered [ 0.310281] acpiphp: Slot [5] registered [ 0.311122] acpiphp: Slot [6] registered [ 0.313122] acpiphp: Slot [7] registered [ 0.314272] acpiphp: Slot [8] registered [ 0.316129] acpiphp: Slot [9] registered [ 0.318114] acpiphp: Slot [10] registered [ 0.319114] acpiphp: Slot [3] registered [ 0.321128] acpiphp: Slot [4] registered [ 0.323198] acpiphp: Slot [11] registered [ 0.325233] acpiphp: Slot [12] registered [ 0.326128] acpiphp: Slot [13] registered [ 0.328115] acpiphp: Slot [14] registered [ 0.330100] acpiphp: Slot [15] registered [ 0.332266] acpiphp: Slot [16] registered [ 0.334238] acpiphp: Slot [17] registered [ 0.336339] acpiphp: Slot [18] registered [ 0.338156] acpiphp: Slot [19] registered [ 0.340126] acpiphp: Slot [20] registered [ 0.341087] acpiphp: Slot [21] registered [ 0.342130] acpiphp: Slot [22] registered [ 0.344156] acpiphp: Slot [23] registered [ 0.345128] acpiphp: Slot [24] registered [ 0.347483] acpiphp: Slot [25] registered [ 0.348148] acpiphp: Slot [26] registered [ 0.350184] acpiphp: Slot [27] registered [ 0.352131] acpiphp: Slot [28] registered [ 0.353289] acpiphp: Slot [29] registered [ 0.355129] acpiphp: Slot [30] registered [ 0.356203] acpiphp: Slot [31] registered [ 0.358122] PCI host bridge to bus 0000:00 [ 0.359023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.363029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.366035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376275] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379780] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.390019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394585] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.398027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.402023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.405795] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409011] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.411056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.414873] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.420026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.433018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.439030] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.445943] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.459037] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528864] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.538085] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.547018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.571020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.584758] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.602022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.615026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.663029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.686622] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.694016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.702025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.731021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.747433] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.783033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.833030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.854056] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.862019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.877021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.896016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.907008] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.909378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.912415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.915472] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.922283] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.927191] iommu: Default domain type: Passthrough [ 0.932468] SCSI subsystem initialized [ 0.934206] ACPI: bus type USB registered [ 0.936170] usbcore: registered new interface driver usbfs [ 0.938113] usbcore: registered new interface driver hub [ 0.940103] usbcore: registered new device driver usb [ 0.942342] pps_core: LinuxPPS API ver. 1 registered [ 0.945015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.948099] PTP clock support registered [ 0.951101] EDAC MC: Ver: 3.0.0 [ 0.953163] PCI: Using ACPI for IRQ routing [ 0.956271] NetLabel: Initializing [ 0.960027] NetLabel: domain hash size = 128 [ 0.963018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.968144] NetLabel: unlabeled traffic allowed by default [ 0.975072] vgaarb: loaded [ 0.977300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.979019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.986463] clocksource: Switched to clocksource kvm-clock [ 1.121783] VFS: Disk quotas dquot_6.6.0 [ 1.126298] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.132640] *** VALIDATE ramfs *** [ 1.136168] *** VALIDATE hugetlbfs *** [ 1.140291] pnp: PnP ACPI init [ 1.147297] pnp: PnP ACPI: found 6 devices [ 1.166190] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.169453] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.171455] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.173432] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.175434] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.177636] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.180212] NET: Registered protocol family 2 [ 1.183152] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.187924] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.191356] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.195777] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.197987] TCP: Hash tables configured (established 65536 bind 65536) [ 1.199650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.201439] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.203119] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.204813] NET: Registered protocol family 1 [ 1.207197] RPC: Registered named UNIX socket transport module. [ 1.210268] RPC: Registered udp transport module. [ 1.212102] RPC: Registered tcp transport module. [ 1.214553] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.217857] NET: Registered protocol family 44 [ 1.219588] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.221517] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.223753] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.226089] PCI: CLS 0 bytes, default 64 [ 1.227874] Unpacking initramfs... [ 2.700172] debug: unmapping init [mem 0xffff8a68bcc54000-0xffff8a68bffbffff] [ 2.707253] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.709757] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.714719] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.218479] Initialise system trusted keyrings [ 3.220150] Key type blacklist registered [ 3.222077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.230838] zbud: loaded [ 3.233963] *** VALIDATE nfs *** [ 3.235222] *** VALIDATE nfs4 *** [ 3.236901] pstore: using deflate compression [ 3.239791] Platform Keyring initialized [ 3.347527] NET: Registered protocol family 38 [ 3.349928] Key type asymmetric registered [ 3.352423] Asymmetric key parser 'x509' registered [ 3.354712] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.358695] io scheduler mq-deadline registered [ 3.360701] io scheduler kyber registered [ 3.362804] io scheduler bfq registered [ 3.364674] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.368108] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.371414] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.374523] ACPI: Power Button [PWRF] [ 3.380172] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.405474] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.423417] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.445226] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.492791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.521965] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.552474] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.563101] Non-volatile memory driver v1.3 [ 3.564898] Linux agpgart interface v0.103 [ 3.600160] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.604795] vda: detected capacity change from 0 to 76775424 [ 3.623000] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.626432] vdb: detected capacity change from 0 to 1073741824 [ 3.641893] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.645075] vdc: detected capacity change from 0 to 2621440000 [ 3.664728] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.668360] vdd: detected capacity change from 0 to 2621440000 [ 3.693625] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.697136] vde: detected capacity change from 0 to 4294967296 [ 3.715142] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.718538] vdf: detected capacity change from 0 to 4294967296 [ 3.728979] libphy: Fixed MDIO Bus: probed [ 3.743018] usbcore: registered new interface driver usbserial_generic [ 3.745576] usbserial: USB Serial support registered for generic [ 3.747764] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.752649] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.754667] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.757689] mousedev: PS/2 mouse device common for all mice [ 3.760593] rtc_cmos 00:05: RTC can wake from S4 [ 3.764166] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.764277] rtc_cmos 00:05: registered as rtc0 [ 3.770640] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.774327] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.776125] intel_pstate: CPU model not supported [ 3.783572] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.789401] hid: raw HID events driver (C) Jiri Kosina [ 3.792446] usbcore: registered new interface driver usbhid [ 3.794794] usbhid: USB HID core driver [ 3.796509] drop_monitor: Initializing network drop monitor service [ 3.798859] Initializing XFRM netlink socket [ 3.800970] NET: Registered protocol family 10 [ 3.804069] Segment Routing with IPv6 [ 3.805765] NET: Registered protocol family 17 [ 3.808059] mpls_gso: MPLS GSO support [ 3.814049] RAS: Correctable Errors collector initialized. [ 3.816189] AVX version of gcm_enc/dec engaged. [ 3.817845] AES CTR mode by8 optimization enabled [ 3.926366] sched_clock: Marking stable (3926270256, 0)->(4907618962, -981348706) [ 3.930294] registered taskstats version 1 [ 3.934317] Loading compiled-in X.509 certificates [ 3.936697] zswap: loaded using pool lzo/zbud [ 3.964202] Key type big_key registered [ 3.978570] Key type encrypted registered [ 3.980612] ima: No TPM chip found, activating TPM-bypass! [ 3.983235] ima: Allocated hash algorithm: sha1 [ 3.985296] ima: No architecture policies found [ 3.987323] evm: Initialising EVM extended attributes: [ 3.989537] evm: security.selinux [ 3.991105] evm: security.ima [ 3.992406] evm: security.capability [ 3.993753] evm: HMAC attrs: 0x1 [ 3.996485] rtc_cmos 00:05: setting system clock to 2026-09-10 04:55:40 UTC (1789016140) [ 4.003906] debug: unmapping init [mem 0xffffffffb3203000-0xffffffffb33fffff] [ 4.007862] debug: unmapping init [mem 0xffffffffb1f82000-0xffffffffb2258fff] [ 4.013169] Write protecting the kernel read-only data: 28672k [ 4.016863] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 4.019513] debug: unmapping init [mem 0xffffffffb0f14000-0xffffffffb0ffffff] [ 4.064625] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.075238] systemd[1]: Detected virtualization kvm. [ 4.077352] systemd[1]: Detected architecture x86-64. [ 4.079816] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.110314] systemd[1]: No hostname configured. [ 4.112260] systemd[1]: Set hostname to . [ 4.114795] random: systemd: uninitialized urandom read (16 bytes read) [ 4.117795] systemd[1]: Initializing machine ID from random generator. [ 4.184755] random: ln: uninitialized urandom read (6 bytes read) [ 4.275241] random: systemd: uninitialized urandom read (16 bytes read) [ 4.277017] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.280342] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.283775] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.039323] device-mapper: uevent: version 1.0.3 [ 5.042167] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.851646] random: fast init done Starting dracut initqueue hook... [ 5.878679] virtio_net virtio0 ens2: renamed from eth0 [ 5.928137] scsi host0: ata_piix [ 5.939988] scsi host1: ata_piix [ 5.942258] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.945077] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.500314] random: crng init done [ 10.501885] random: 7 urandom warning(s) missed due to ratelimiting [ 10.503429] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.125196] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.329489] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.538533] SELinux: Disabled at runtime. [ 12.598413] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.605602] systemd[1]: Detected virtualization kvm. [ 12.607546] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.064743] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.067300] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.071871] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.077282] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.080295] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.087598] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.092553] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 13.279946] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.572112] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.938562] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.961704] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.123868] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.133181] EDAC sbridge: Ver: 1.1.2 [ 15.454607] Key type dns_resolver registered [ 15.755801] NFS: Registering the id_resolver key type [ 15.758449] Key type id_resolver registered [ 15.761202] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 47.119345] libcfs: loading out-of-tree module taints kernel. [ 47.284976] Key type ._llcrypt registered [ 47.287354] Key type .llcrypt registered [ 47.465761] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 69.413214] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 70.744705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.760134] alg: No test for adler32 (adler32-zlib) [ 73.027198] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 75.065151] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 76.856485] Key type lgssc registered [ 79.389112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.388436] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.123581] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 157.214459] hrtimer: interrupt took 4267642 ns [ 163.638195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 163.697277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.098492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.160418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.255323] Lustre: lustre-MDT0000: new disk, initializing [ 165.371112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.399580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.927817] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.052054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.183365] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.216448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.222855] Lustre: Skipped 1 previous similar message [ 184.330600] Lustre: lustre-MDT0001: new disk, initializing [ 184.398759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.436851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.453993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.625307] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.333479] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.011786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.159449] Lustre: lustre-OST0000: new disk, initializing [ 204.162514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.166623] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.199547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.487617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.506597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.586552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.159727] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.623493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.800408] Lustre: lustre-OST0001: new disk, initializing [ 225.808262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.818153] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.918042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.361714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.560967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.577827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.615637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.420322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.886232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.362269] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.964982] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 268.525097] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 270.926186] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 273.347962] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 275.024947] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 01:00:10 EDT 2026 [ 293.286689] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.675867] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.295920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.306628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.326769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.384721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.400687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.032714] Lustre: server umount lustre-MDT0000 complete [ 322.727634] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789016459 with bad export cookie 16284354492985132210 [ 322.732448] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.742608] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.623834] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.624516] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.626831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.626840] Lustre: Skipped 3 previous similar messages [ 326.638556] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 326.670187] Lustre: Skipped 3 previous similar messages [ 327.649269] LustreError: 7691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.667183] LustreError: 7691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 332.771687] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.771880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.806654] Lustre: Skipped 4 previous similar messages [ 337.375140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.652409] Lustre: server umount lustre-MDT0001 complete [ 343.083540] Lustre: server umount lustre-OST0000 complete [ 346.740383] Lustre: server umount lustre-OST0001 complete [ 355.007177] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 364.368300] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 406.760556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.093214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 407.141653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.217546] Lustre: lustre-MDT0000: new disk, initializing [ 407.294967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.312551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.858503] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 425.380960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.574417] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.697587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.707082] Lustre: Skipped 1 previous similar message [ 425.818413] Lustre: lustre-MDT0001: new disk, initializing [ 426.053814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.097802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.119435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.155597] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 441.711134] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 449.602417] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 458.407045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 458.929515] Lustre: lustre-OST0000: new disk, initializing [ 458.953487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 458.969457] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 459.132214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.250453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 460.265464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 460.559922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.558222] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 477.874791] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.083899] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.341118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.467572] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.731303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.098128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.115636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.145201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.494448] Lustre: server umount lustre-OST0000 complete [ 506.851791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.856255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.862576] Lustre: Skipped 2 previous similar messages [ 506.875455] Lustre: Skipped 2 previous similar messages [ 512.668927] Lustre: server umount lustre-MDT0000 complete [ 517.093836] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.124760] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 517.485287] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789016653 with bad export cookie 16284354492985133743 [ 517.489273] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.499920] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 518.127288] Lustre: server umount lustre-MDT0001 complete [ 523.463826] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:04:18 (1789016658) [ 527.354519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 529.718890] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:04:24 (1789016664) [ 537.262150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.935227] LustreError: 19462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.043880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.506084] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 548.330312] LustreError: 19463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.354492] LustreError: 19463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 554.819833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.168492] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 570.082896] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.926761] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.902157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.747479] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 599.602988] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 608.757318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 608.776236] LustreError: Skipped 1 previous similar message [ 608.786303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.814176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.816612] Lustre: Skipped 2 previous similar messages [ 614.013563] Lustre: server umount lustre-OST0000 complete [ 633.312045] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 633.520982] Lustre: server umount lustre-MDT0000 complete [ 635.363797] LustreError: 20400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.398526] LustreError: 20400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 637.631589] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789016774 with bad export cookie 16284354492985134625 [ 637.635473] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.641215] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 638.002263] Lustre: server umount lustre-MDT0001 complete [ 651.820402] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 655.819529] Key type lgssc unregistered [ 656.161538] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.170497] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.190528] LNet: Removed LNI 192.168.202.138@tcp [ 658.743266] Key type .llcrypt unregistered [ 658.745789] Key type ._llcrypt unregistered [ 685.043260] Key type ._llcrypt registered [ 685.053756] Key type .llcrypt registered [ 685.241089] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 686.498395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.660110] alg: No test for adler32 (adler32-zlib) [ 688.485650] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 688.759928] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 690.471212] Key type lgssc registered [ 691.922414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.179537] loop: module loaded [ 700.281070] loop0: detected capacity change from 0 to 204800000 [ 700.573115] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.612363] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.317646] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.608951] loop0: detected capacity change from 0 to 204800000 [ 708.900220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.937359] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.967762] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.114762] loop0: detected capacity change from 0 to 204800000 [ 721.116829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 721.137445] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.581308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 722.648333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 722.764455] Lustre: lustre-MDT0000: new disk, initializing [ 722.970610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.010319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.630495] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 736.489174] loop1: detected capacity change from 0 to 204800000 [ 742.503141] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 742.985348] Lustre: lustre-OST07c6: new disk, initializing [ 742.993042] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 743.001274] Lustre: Skipped 1 previous similar message [ 743.007308] Lustre: 26189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 743.110659] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 749.331332] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 749.708311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 749.720359] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 749.755175] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 754.811857] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 754.827091] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 754.832532] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 754.840749] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 754.849027] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 754.860499] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 775.692864] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 775.707821] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.720485] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.728214] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 775.742778] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 775.752679] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.714818] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 780.719620] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 780.728102] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.742546] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.751318] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.761671] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.773350] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 780.784363] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.795286] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 780.800580] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.812093] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.815912] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.009307] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 784.017706] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 784.023505] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 784.030579] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.039157] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 784.044916] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.052325] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 784.058597] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.076975] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 784.085790] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.092882] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 784.098595] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.207443] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 789.214143] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 789.220300] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.224765] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.232179] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.238782] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.245086] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.253399] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.261551] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.267433] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.274106] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.279149] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.564969] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 797.590737] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 797.600828] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 797.612065] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.619393] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.640438] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.652184] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 797.674633] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.695653] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 797.705747] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.713995] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.729242] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.055847] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 816.067223] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 816.076039] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 816.099964] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 816.120663] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.132765] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 816.155555] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 816.169907] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 816.183722] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 816.196430] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 816.219177] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.227700] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 857.067261] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.095319] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 859.697857] Lustre: server umount lustre-OST07c6 complete [ 869.212576] Lustre: server umount lustre-MDT0000 complete [ 884.640027] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 887.236918] Key type lgssc unregistered [ 887.556822] LNet: 28751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.564479] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.619162] LNet: Removed LNI 192.168.202.138@tcp [ 889.415285] Key type .llcrypt unregistered [ 889.419066] Key type ._llcrypt unregistered [ 907.839660] Key type ._llcrypt registered [ 907.841740] Key type .llcrypt registered [ 907.937813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 909.749866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.835586] alg: No test for adler32 (adler32-zlib) [ 911.098805] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 911.504198] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 913.231407] Key type lgssc registered [ 914.428237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.429969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 922.445178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.063100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.307532] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 934.370400] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.402822] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 936.808944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.484659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.243392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 950.454641] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 956.504606] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 962.900570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.295322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.491252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 969.939175] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 978.355234] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 997.350666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.368848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.376232] Lustre: Skipped 1 previous similar message [ 998.435296] Lustre: server umount lustre-OST0000 complete [ 1002.466213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.468512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.483186] Lustre: Skipped 2 previous similar messages [ 1002.509383] Lustre: Skipped 1 previous similar message [ 1007.588200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.591034] Lustre: Skipped 1 previous similar message [ 1012.718452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.722387] Lustre: Skipped 1 previous similar message [ 1016.288124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1016.488210] Lustre: server umount lustre-MDT0000 complete [ 1016.801802] LustreError: 30017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.825580] LustreError: 30017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1020.150987] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789017156 with bad export cookie 10945570637096401119 [ 1020.161764] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.603204] Lustre: server umount lustre-MDT0001 complete [ 1032.567059] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1035.613754] Key type lgssc unregistered [ 1035.947926] LNet: 33754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.973943] LNetError: 33754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.996453] LNet: Removed LNI 192.168.202.138@tcp [ 1036.790171] Key type .llcrypt unregistered [ 1036.792201] Key type ._llcrypt unregistered [ 1054.893802] Key type ._llcrypt registered [ 1054.896938] Key type .llcrypt registered [ 1054.992816] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1056.682917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.738196] alg: No test for adler32 (adler32-zlib) [ 1057.757288] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1058.007388] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1059.695216] Key type lgssc registered [ 1061.238720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.616790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1068.652059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.303139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.645979] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1079.789324] LustreError: 35021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.824458] LustreError: 35021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1083.408976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.843856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1088.575322] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1096.512458] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1103.327428] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.116143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.390486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.530461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1118.555279] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1126.207938] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1141.216510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.221100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.229088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.290763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.294442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.296988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.684979] Lustre: server umount lustre-OST0000 complete [ 1154.535511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.548956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.560557] Lustre: Skipped 1 previous similar message [ 1154.583641] Lustre: Skipped 2 previous similar messages [ 1157.884754] Lustre: server umount lustre-MDT0000 complete [ 1159.652563] LustreError: 35015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.666639] LustreError: 35015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1161.856631] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789017298 with bad export cookie 15451348180057774220 [ 1161.858799] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.882393] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1162.410371] Lustre: server umount lustre-MDT0001 complete [ 1174.070865] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1177.167855] Key type lgssc unregistered [ 1177.487171] LNet: 38581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.492259] LNetError: 38581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.528551] LNet: Removed LNI 192.168.202.138@tcp [ 1178.391276] Key type .llcrypt unregistered [ 1178.393431] Key type ._llcrypt unregistered [ 1200.642461] Key type ._llcrypt registered [ 1200.646941] Key type .llcrypt registered [ 1200.741441] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1201.659939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1201.675443] alg: No test for adler32 (adler32-zlib) [ 1202.908975] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1203.251121] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1204.999229] Key type lgssc registered [ 1205.850901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1213.921341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1213.938654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.487333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.925189] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1224.674899] LustreError: 39972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.688467] LustreError: 39972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1229.792422] LustreError: 39973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.277193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.558425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.169981] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1242.146714] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1247.889171] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.028745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.317201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.481430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1262.294371] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1269.515762] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.875246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.878494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.887669] Lustre: Skipped 1 previous similar message [ 1275.902668] Lustre: Skipped 2 previous similar messages [ 1277.534095] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 1278.328302] Lustre: server umount lustre-MDT0000 complete [ 1280.994599] LustreError: 39972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.023188] LustreError: 39972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1282.050388] LustreError: 39953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789017418 with bad export cookie 10489022240010614236 [ 1282.061595] LustreError: 39953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1282.066124] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.661763] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 1286.112715] LustreError: 42317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.123317] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.132297] LustreError: 42317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1286.159129] Lustre: Skipped 1 previous similar message [ 1286.167695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1288.583688] Lustre: server umount lustre-MDT0001 complete [ 1317.343160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1317.655419] Lustre: server umount lustre-OST0000 complete [ 1341.281689] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1345.330425] Key type lgssc unregistered [ 1345.708583] LNet: 43836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.715514] LNetError: 43836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.734600] LNet: Removed LNI 192.168.202.138@tcp [ 1346.922234] Key type .llcrypt unregistered [ 1346.927153] Key type ._llcrypt unregistered [ 1374.311645] Key type ._llcrypt registered [ 1374.317320] Key type .llcrypt registered [ 1374.435945] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1375.613293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1375.640810] alg: No test for adler32 (adler32-zlib) [ 1376.932500] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1377.089358] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1378.695292] Key type lgssc registered [ 1379.883217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1387.620308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1387.639432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.266280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.202940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1398.762979] LustreError: 45249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.781577] LustreError: 45249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1401.882351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.239267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1406.968598] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1415.664358] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1422.818714] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.837213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1431.112721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.255036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1437.422310] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1445.247409] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.007054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.011167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.044541] Lustre: Skipped 1 previous similar message [ 1457.135545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1457.149109] Lustre: Skipped 2 previous similar messages [ 1457.223858] LustreError: 47072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.388842] Lustre: server umount lustre-OST0000 complete [ 1477.600604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.617912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.628781] Lustre: Skipped 1 previous similar message [ 1481.769829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.785829] Lustre: Skipped 1 previous similar message [ 1482.116651] Lustre: server umount lustre-MDT0000 complete [ 1486.823843] LustreError: 47738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.865318] LustreError: 47738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1488.091785] LustreError: 45231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789017624 with bad export cookie 11360985167593129794 [ 1488.121553] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.558813] Lustre: server umount lustre-MDT0001 complete [ 1505.542940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1508.591176] Key type lgssc unregistered [ 1508.943092] LNet: 48958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.961233] LNetError: 48958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.998387] LNet: Removed LNI 192.168.202.138@tcp [ 1511.092134] Key type .llcrypt unregistered [ 1511.094375] Key type ._llcrypt unregistered [ 1536.453582] Key type ._llcrypt registered [ 1536.458047] Key type .llcrypt registered [ 1536.640812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1538.220169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1538.492579] alg: No test for adler32 (adler32-zlib) [ 1539.686455] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1540.034958] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1541.815419] Key type lgssc registered [ 1543.465174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1552.757366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1552.776829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.632924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.616908] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1565.162641] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.181385] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1570.278504] LustreError: 50370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.825544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.451632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1578.232645] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1588.728389] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1597.826477] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1607.774136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.266828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.926409] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1623.511961] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1625.735689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1633.208897] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1636.752554] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1649.479644] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-10 1h23m03s [ 1651.066231] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1654.242926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1654.262956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.301333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.897368] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 1654.909282] Lustre: Skipped 2 previous similar messages [ 1655.942131] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 1655.953129] Lustre: Skipped 3 previous similar messages [ 1659.611345] Lustre: server umount lustre-MDT0000 complete [ 1661.003594] LustreError: 50366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.434288] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1667.163205] LustreError: 50366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.186216] LustreError: 50366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1670.620746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.783274] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.163330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.263564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1672.179597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1675.824061] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1679.343377] Lustre: DEBUG MARKER: Wait for df (36710) ... [ 1680.937262] Lustre: DEBUG MARKER: done [ 1686.497626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1686.506068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.521923] Lustre: Skipped 2 previous similar messages [ 1686.534088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.560706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.644181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.658062] Lustre: Skipped 3 previous similar messages [ 1700.320264] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1700.582138] Lustre: server umount lustre-OST0000 complete [ 1708.015234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.015623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1708.032147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1708.060221] Lustre: Skipped 3 previous similar messages [ 1712.117590] Lustre: server umount lustre-MDT0000 complete [ 1715.695272] LustreError: 50365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.712462] LustreError: 50365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1716.636509] LustreError: 50349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789017853 with bad export cookie 8046964773200717728 [ 1716.649250] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.992576] Lustre: server umount lustre-MDT0001 complete [ 1730.308277] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1733.342963] Key type lgssc unregistered [ 1733.735768] LNet: 55297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.750660] LNetError: 55297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.785858] LNet: Removed LNI 192.168.202.138@tcp [ 1734.998391] Key type .llcrypt unregistered [ 1735.001307] Key type ._llcrypt unregistered [ 1780.958960] Key type ._llcrypt registered [ 1780.972278] Key type .llcrypt registered [ 1781.141850] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1782.847690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.955528] alg: No test for adler32 (adler32-zlib) [ 1784.495916] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1784.845342] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1786.553491] Key type lgssc registered [ 1789.462129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.747632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1799.779820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.068832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1801.083122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1801.196899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.933629] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1815.422333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.534599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1815.575530] Lustre: 57503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1815.954366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.475328] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1828.189854] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1834.620752] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1841.989395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.281161] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1842.497517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.067734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1850.809315] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1858.895833] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1866.570243] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-10 1h26m41s [ 1868.469810] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1875.488550] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1877.983804] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1878.016848] LustreError: 58781:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1898.102931] Lustre: lustre-MDT0000: Client c7074f56-a6ca-4303-96ad-400e4a6a0636 (at 192.168.202.38@tcp) reconnecting [ 1918.039216] LustreError: 58781:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1920.364171] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1922.889393] Lustre: DEBUG MARKER: done [ 1930.217276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1930.225496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.247146] Lustre: Skipped 1 previous similar message [ 1930.251097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1930.260523] Lustre: Skipped 1 previous similar message [ 1930.762251] Lustre: server umount lustre-OST0000 complete [ 1935.329253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.336645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.351334] Lustre: Skipped 1 previous similar message [ 1935.361696] Lustre: Skipped 1 previous similar message [ 1936.417713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1936.434116] Lustre: Skipped 1 previous similar message [ 1941.464965] Lustre: server umount lustre-MDT0000 complete [ 1945.263877] LustreError: 58244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018081 with bad export cookie 15862311110038108008 [ 1945.269294] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.283931] LustreError: 58244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1945.590986] Lustre: server umount lustre-MDT0001 complete [ 1956.644318] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1959.492591] Key type lgssc unregistered [ 1959.925611] LNet: 61832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.936181] LNetError: 61832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.987236] LNet: Removed LNI 192.168.202.138@tcp [ 1961.327432] Key type .llcrypt unregistered [ 1961.329408] Key type ._llcrypt unregistered [ 2009.878643] Key type ._llcrypt registered [ 2009.880612] Key type .llcrypt registered [ 2009.984725] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2011.327084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.374131] alg: No test for adler32 (adler32-zlib) [ 2012.378354] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2012.526614] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2014.151318] Key type lgssc registered [ 2015.373903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.565206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2017.582440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.833915] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2018.849620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2019.052189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.920928] Lustre: Failing over lustre-MDT0000 [ 2020.307415] Lustre: server umount lustre-MDT0000 complete [ 2026.958413] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:29:22 (1789018162) [ 2034.080988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.646683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.755470] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2048.516068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.591128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2048.618918] Lustre: 64635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2048.896681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.795498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2061.902410] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2069.137603] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2077.726774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.177693] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2078.502551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.480714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2087.598942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2094.402529] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.227049] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2108.385929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.398386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.411062] Lustre: Skipped 2 previous similar messages [ 2108.438710] Lustre: Skipped 2 previous similar messages [ 2111.931053] Lustre: server umount lustre-MDT0000 complete [ 2115.729625] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018252 with bad export cookie 14837186856597680940 [ 2115.740823] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.750696] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2116.269816] Lustre: server umount lustre-MDT0001 complete [ 2119.782079] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2125.418635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2135.023729] Lustre: 63558:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018255/real 1789018255] req@00000000bbc1ae96 x1875921494648064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.083149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.340068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.055368] Lustre: 63557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018257/real 1789018257] req@0000000090dd2e68 x1875921494648192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.193783] Lustre: 63557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018260/real 1789018260] req@00000000a933c505 x1875921494648320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.893110] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.923500] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2142.046264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.192317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2143.011531] Lustre: 63558:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018262/real 1789018262] req@000000004f8c4ce1 x1875921494648576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.035431] LustreError: 66487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.063072] LustreError: 66487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2145.071640] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.100664] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2147.103203] Lustre: 63558:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018267/real 1789018267] req@000000006bf5c302 x1875921494648704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2147.435414] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2150.370486] LustreError: 68410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.395879] LustreError: 68410:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2155.493282] LustreError: 68410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.522433] LustreError: 68410:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2157.094887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.890259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.188282] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.287432] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.887856] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2173.187024] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.198856] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2188.257441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.270344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.283762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.798608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.809111] Lustre: server umount lustre-MDT0000 complete [ 2194.922625] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.940552] LustreError: 66486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2198.611604] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018335 with bad export cookie 14837186856597683222 [ 2198.615744] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.617060] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2198.898306] Lustre: server umount lustre-MDT0001 complete [ 2208.282314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2216.415182] Lustre: 63557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789018336/real 1789018336] req@000000007a06cbe0 x1875921494678400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789018352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2216.444478] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.457511] Lustre: Skipped 2 previous similar messages [ 2219.027683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.648356] LustreError: 63555:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8b96006 x1875921494679424/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2223.932503] LustreError: 69144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.042485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.115352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2228.325861] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2237.404179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.058249] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2243.627691] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2245.160845] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2252.681121] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2259.235539] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2266.343427] Lustre: server umount lustre-MDT0000 complete [ 2268.649369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2268.666039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.692282] LustreError: 66487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.720327] LustreError: 66487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2271.446743] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018407 with bad export cookie 14837186856597684601 [ 2271.453482] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.459819] LustreError: 67213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2271.947235] Lustre: server umount lustre-MDT0001 complete [ 2279.643149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2283.211553] Lustre: DEBUG MARKER: files compared the same [ 2295.764714] Lustre: server umount lustre-OST0000 complete [ 2315.373191] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2318.917505] Key type lgssc unregistered [ 2319.269642] LNet: 74370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.283620] LNetError: 74370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.357977] LNet: Removed LNI 192.168.202.138@tcp [ 2321.391896] Key type .llcrypt unregistered [ 2321.393553] Key type ._llcrypt unregistered [ 2347.235486] Key type ._llcrypt registered [ 2347.240188] Key type .llcrypt registered [ 2347.325844] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2348.747936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.785173] alg: No test for adler32 (adler32-zlib) [ 2349.897766] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2350.150728] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2351.824395] Key type lgssc registered [ 2352.936910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2361.455647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2361.492675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.160485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.507536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2372.589186] LustreError: 75783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.619254] LustreError: 75783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2377.708254] LustreError: 75782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.512296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.976122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2385.105317] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2393.128883] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.290093] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2405.975278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.295817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.363260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2412.949952] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug +malloc all [ 2421.156642] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.065389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.068788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.075525] Lustre: Skipped 1 previous similar message [ 2436.085554] Lustre: Skipped 1 previous similar message [ 2440.459761] Lustre: server umount lustre-OST0000 complete [ 2446.303943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.325655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.330686] Lustre: Skipped 1 previous similar message [ 2450.756949] Lustre: server umount lustre-MDT0000 complete [ 2451.426520] LustreError: 75782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.438660] LustreError: 75782:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2454.259584] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018590 with bad export cookie 8996755090252996297 [ 2454.277471] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.787763] Lustre: server umount lustre-MDT0001 complete [ 2465.739734] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2468.968989] Key type lgssc unregistered [ 2469.226742] LNet: 79334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.229996] LNetError: 79334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.249249] LNet: Removed LNI 192.168.202.138@tcp [ 2469.923184] Key type .llcrypt unregistered [ 2469.931800] Key type ._llcrypt unregistered [ 2493.571380] Key type ._llcrypt registered [ 2493.576096] Key type .llcrypt registered [ 2493.697367] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2494.920405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.982704] alg: No test for adler32 (adler32-zlib) [ 2496.201652] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2496.521708] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2498.199171] Key type lgssc registered [ 2499.354444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.339403] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2506.374909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.079694] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2527.201753] LustreError: 80722:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 2553.887657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.994582] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2570.999657] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2580.332474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.515191] Lustre: *** cfs_fail_loc=706, val=0*** [ 2581.106934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.238080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2586.094857] LustreError: 80748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.128630] LustreError: 80748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2586.296356] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2589.223098] LustreError: 81812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.252197] LustreError: 81812:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2594.277425] LustreError: 81814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.299548] LustreError: 81814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2595.735733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.329315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.072969] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2608.769119] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2614.592898] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2621.923244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2621.923739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.938782] LustreError: Skipped 1 previous similar message [ 2621.941125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2622.007031] Lustre: Skipped 1 previous similar message [ 2627.052630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.056243] Lustre: Skipped 1 previous similar message [ 2631.137228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2631.357540] Lustre: server umount lustre-OST0000 complete [ 2637.280773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.284054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.311928] Lustre: Skipped 1 previous similar message [ 2637.345418] Lustre: Skipped 2 previous similar messages [ 2641.266154] Lustre: server umount lustre-MDT0000 complete [ 2642.415700] LustreError: 81815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.440140] LustreError: 81815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2645.745109] LustreError: 80728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789018782 with bad export cookie 2679310276845231733 [ 2645.759544] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.762803] LustreError: 80728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2646.165102] Lustre: server umount lustre-MDT0001 complete [ 2658.293889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 2661.432609] Key type lgssc unregistered [ 2661.744668] LNet: 84343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.751283] LNetError: 84343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2661.789738] LNet: Removed LNI 192.168.202.138@tcp [ 2662.690366] Key type .llcrypt unregistered [ 2662.692171] Key type ._llcrypt unregistered [ 2688.967935] Key type ._llcrypt registered [ 2688.970771] Key type .llcrypt registered [ 2689.079423] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2690.810155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.867271] alg: No test for adler32 (adler32-zlib) [ 2691.973726] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2692.231616] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 2693.919338] Key type lgssc registered [ 2695.052782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2703.492393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2703.526420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.660112] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2716.056811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.555530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.754277] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2721.766492] LustreError: 86444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.786621] LustreError: 86444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2725.859651] LustreError: 86447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.881982] LustreError: 86447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2729.015257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.466171] LustreError: 86444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.497291] LustreError: 86444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2729.508222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.563857] LustreError: 86448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.594851] LustreError: 86448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2735.577220] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2739.686330] LustreError: 86447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.715150] LustreError: 86447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2744.459376] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.478155] Lustre: MGS: Client fbe8fb38-6826-436f-bb16-350cec319c97 (at 0@lo) reconnecting [ 2744.517426] LustreError: 86458:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2744.815275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.887474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2749.273632] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2765.285381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.295313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.304628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.322767] Lustre: Skipped 1 previous similar message [ 2765.343664] Lustre: Skipped 1 previous similar message [ 2770.000787] Lustre: server umount lustre-OST0000 complete [ 2775.521332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.525655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.546128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.569329] Lustre: Skipped 1 previous similar message [ 2779.901747] Lustre: server umount lustre-MDT0000 complete [ 2780.640981] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.662096] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2784.302476] Lustre: server umount lustre-MDT0001 complete [ 2787.832841] Lustre: server umount MGS complete [ 2799.322266] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:42:14 (1789018934) [ 2830.938534] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2841.285965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2879.079540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.112863] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2894.532898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.877548] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2894.960799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2895.087116] Lustre: lustre-MDT0001: new disk, initializing [ 2895.187539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.225607] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2895.242775] Lustre: 93634:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2895.258412] LustreError: 93634:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2899.776504] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2910.579542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.909395] Lustre: lustre-OST0000: new disk, initializing [ 2910.913338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2910.919405] Lustre: 94399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2911.037631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.559365] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2920.032199] Lustre: 94823:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f5608439 x1875922206629632/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789019072 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2927.903142] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2934.632279] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.642208] Lustre: MGS: Client 885aa6bb-368f-4198-8fe1-5cef674fdd6e (at 0@lo) reconnecting [ 2934.692292] Lustre: 95444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2934.704200] LustreError: 93620:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2934.714391] LustreError: 93620:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2934.724862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2934.802329] Lustre: lustre-MDT0000: new disk, initializing [ 2934.945185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.971131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2934.993967] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2936.040088] Lustre: 95469:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000007f2a2119 x1875922206636288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789019088 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2939.934549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2941.284025] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2945.126590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2945.143829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2945.378990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2971.105264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.118286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.139788] Lustre: Skipped 1 previous similar message [ 2971.149656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.156960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.166911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.195380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.202781] Lustre: server umount lustre-OST0000 complete [ 2984.417721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.425461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2984.428779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.428796] Lustre: Skipped 2 previous similar messages [ 2984.445356] Lustre: Skipped 1 previous similar message [ 2986.264286] Lustre: server umount lustre-MDT0000 complete [ 2995.901810] Lustre: server umount lustre-MDT0001 complete [ 2999.750372] Lustre: server umount MGS complete [ 3011.687095] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:45:46 (1789019146) [ 3031.345450] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3034.704286] Key type lgssc unregistered [ 3035.010816] LNet: 97965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.014153] LNetError: 97965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.026735] LNet: Removed LNI 192.168.202.138@tcp [ 3036.102516] Key type .llcrypt unregistered [ 3036.105179] Key type ._llcrypt unregistered [ 3053.555814] Key type ._llcrypt registered [ 3053.560132] Key type .llcrypt registered [ 3053.715711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3054.665194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3054.744412] alg: No test for adler32 (adler32-zlib) [ 3055.926462] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3056.196930] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3057.888435] Key type lgssc registered [ 3059.102727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.056539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3068.074442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.463232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.675763] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3078.626839] LustreError: 99181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.642429] LustreError: 99181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3082.552312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.959463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.249751] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3095.116413] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3101.471261] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3108.723895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.082281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.271509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3116.235321] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3125.194511] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3141.099813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.118369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.148704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.141642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3143.158636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.168399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.201666] Lustre: Skipped 1 previous similar message [ 3145.814512] Lustre: server umount lustre-OST0000 complete [ 3153.381297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.395776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.399122] Lustre: Skipped 1 previous similar message [ 3153.423833] Lustre: Skipped 1 previous similar message [ 3156.920072] Lustre: server umount lustre-MDT0000 complete [ 3158.496605] LustreError: 99181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.504232] LustreError: 99181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3161.211841] LustreError: 99163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019297 with bad export cookie 10258226192604662510 [ 3161.235511] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.599213] Lustre: server umount lustre-MDT0001 complete [ 3173.639432] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3176.998805] Key type lgssc unregistered [ 3177.377950] LNet: 102732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.389520] LNetError: 102732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.404550] LNet: Removed LNI 192.168.202.138@tcp [ 3178.247345] Key type .llcrypt unregistered [ 3178.249104] Key type ._llcrypt unregistered [ 3195.450452] Key type ._llcrypt registered [ 3195.452965] Key type .llcrypt registered [ 3195.565960] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3196.724529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.781336] alg: No test for adler32 (adler32-zlib) [ 3198.098431] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3198.365778] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3200.081729] Key type lgssc registered [ 3201.616644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3208.514675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3208.522283] LustreError: 103811:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3208.536685] LustreError: 103813:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3208.542096] LustreError: 103811:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3208.549648] LustreError: 103811:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3208.559545] LustreError: 103811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3208.565404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.204442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.587298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.803588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.062265] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3229.126765] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lsmod [ 3232.080413] LustreError: 105013:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3232.108779] LustreError: 105020:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3232.114941] LustreError: 105013:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3232.120565] LustreError: 105013:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3232.128844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.131551] LustreError: 105013:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3232.479927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.563596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3239.408182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3239.423917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.455859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.913812] Lustre: server umount lustre-MDT0000 complete [ 3244.024912] LustreError: 103850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.052465] LustreError: 103850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3247.393732] LustreError: 103826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019383 with bad export cookie 10545777879451135322 [ 3247.398311] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.402143] LustreError: 103826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3248.223285] LustreError: 104282:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005bcb9f15 x1875922737190272/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3248.477734] Lustre: server umount lustre-MDT0001 complete [ 3261.723702] Lustre: server umount lustre-OST0000 complete [ 3270.682877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.141713] LustreError: 106085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.221727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.263063] LustreError: 106086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.696111] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3281.378910] LustreError: 106085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.502731] LustreError: 106086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.366319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.768863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3292.716104] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3301.951748] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3306.961222] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3313.727309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.206414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.290787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3320.579655] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3327.931317] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3340.779119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3340.784400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.795280] Lustre: Skipped 2 previous similar messages [ 3340.817609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.830388] Lustre: Skipped 2 previous similar messages [ 3342.816841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.825948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.827535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.845665] Lustre: Skipped 1 previous similar message [ 3344.554526] Lustre: server umount lustre-OST0000 complete [ 3349.473740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3349.486201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.508223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.428967] Lustre: server umount lustre-MDT0000 complete [ 3358.148222] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019494 with bad export cookie 10545777879451136267 [ 3358.150235] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.158454] LustreError: 106067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3358.636630] Lustre: server umount lustre-MDT0001 complete [ 3370.488369] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3372.955090] Key type lgssc unregistered [ 3373.229236] LNet: 109618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.237704] LNetError: 109618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3373.253673] LNet: Removed LNI 192.168.202.138@tcp [ 3374.320133] Key type .llcrypt unregistered [ 3374.323430] Key type ._llcrypt unregistered [ 3409.485877] Key type ._llcrypt registered [ 3409.490718] Key type .llcrypt registered [ 3409.652731] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3410.392586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3410.446485] alg: No test for adler32 (adler32-zlib) [ 3411.750201] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3412.187352] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3414.015334] Key type lgssc registered [ 3414.987301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3423.068369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3423.093364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.622540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.594624] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3434.993501] LustreError: 111036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.016388] LustreError: 111036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3439.340981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.777346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3444.341961] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3453.183513] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3459.942982] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3467.252287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.666056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.698616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3475.711354] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3486.085913] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3491.989989] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3495.487518] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3505.640169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3505.641300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.647041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.653402] LustreError: Skipped 1 previous similar message [ 3505.699866] Lustre: Skipped 1 previous similar message [ 3506.831866] Lustre: server umount lustre-OST0000 complete [ 3510.759526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.764236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.787975] Lustre: Skipped 1 previous similar message [ 3510.805202] Lustre: Skipped 2 previous similar messages [ 3515.873846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.880645] Lustre: Skipped 1 previous similar message [ 3516.905575] Lustre: server umount lustre-MDT0000 complete [ 3520.285977] LustreError: 113565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019656 with bad export cookie 11773435266617991406 [ 3520.291026] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.299982] LustreError: 113565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.578569] Lustre: server umount lustre-MDT0001 complete [ 3532.011221] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3534.550378] Key type lgssc unregistered [ 3534.776845] LNet: 114667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3534.781398] LNetError: 114667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3535.856663] LNet: Removed LNI 192.168.202.138@tcp [ 3537.219262] Key type .llcrypt unregistered [ 3537.220755] Key type ._llcrypt unregistered [ 3555.300613] Key type ._llcrypt registered [ 3555.306666] Key type .llcrypt registered [ 3555.491378] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3557.111973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3557.342053] alg: No test for adler32 (adler32-zlib) [ 3558.516968] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3558.826504] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3560.527544] Key type lgssc registered [ 3561.666616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3569.964601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3569.985644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.608703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.724763] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3581.929583] LustreError: 115888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3581.943204] LustreError: 115888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3586.017854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.417561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3591.848281] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3600.816846] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3606.915545] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3614.533647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3614.818040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.891431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3620.967497] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3628.892365] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3642.848834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3642.865653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.899388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3646.945871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.967888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.970140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.015449] Lustre: Skipped 1 previous similar message [ 3648.575972] Lustre: server umount lustre-OST0000 complete [ 3653.090170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3653.102473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.119312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.184628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.205053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.014391] Lustre: server umount lustre-MDT0000 complete [ 3662.306863] LustreError: 115881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.319204] LustreError: 115881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3662.828183] LustreError: 115866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019799 with bad export cookie 5498018430791679920 [ 3662.834289] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.841098] LustreError: 115866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3663.363925] Lustre: server umount lustre-MDT0001 complete [ 3674.922334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3677.524867] Key type lgssc unregistered [ 3677.789681] LNet: 119444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.800365] LNetError: 119444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.831737] LNet: Removed LNI 192.168.202.138@tcp [ 3679.194889] Key type .llcrypt unregistered [ 3679.196809] Key type ._llcrypt unregistered [ 3703.811570] Key type ._llcrypt registered [ 3703.813883] Key type .llcrypt registered [ 3703.918268] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3704.851590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3704.885656] alg: No test for adler32 (adler32-zlib) [ 3705.934500] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3706.253065] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3707.959527] Key type lgssc registered [ 3709.308934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3717.570189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3717.582971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.043536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.064682] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3729.377734] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.409551] LustreError: 120860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3731.834796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.201500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3736.143648] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3743.184355] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.153806] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3756.082263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.426308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.857553] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3763.581359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3769.503674] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.438122] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3779.855721] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3779.861462] Lustre: Skipped 2 previous similar messages [ 3783.835311] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3783.842327] Lustre: Skipped 2 previous similar messages [ 3785.234717] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3785.271038] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3795.101142] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3795.107236] Lustre: Skipped 2 previous similar messages [ 3803.194660] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.38@tcp 192.168.0.[2,10]@tcp [ 3806.540780] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.38/24@tcp in log lustre-MDT0000 [ 3806.546196] Lustre: Skipped 2 previous similar messages [ 3811.399165] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 3811.410560] Lustre: Skipped 1 previous similar message [ 3823.020325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.021212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.030459] Lustre: Skipped 1 previous similar message [ 3824.099715] Lustre: server umount lustre-OST0000 complete [ 3828.202281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.221510] Lustre: Skipped 1 previous similar message [ 3828.227814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.239422] Lustre: Skipped 2 previous similar messages [ 3833.313545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.326834] Lustre: Skipped 1 previous similar message [ 3834.034192] Lustre: server umount lustre-MDT0000 complete [ 3837.820800] LustreError: 123489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789019974 with bad export cookie 7694265241329837041 [ 3837.822616] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3837.833142] LustreError: 123489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3838.107596] Lustre: server umount lustre-MDT0001 complete [ 3851.579343] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 3855.072508] Key type lgssc unregistered [ 3855.512658] LNet: 125741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3855.530552] LNetError: 125741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3855.545839] LNet: Removed LNI 192.168.202.138@tcp [ 3856.730375] Key type .llcrypt unregistered [ 3856.733822] Key type ._llcrypt unregistered [ 3892.754632] Key type ._llcrypt registered [ 3892.759934] Key type .llcrypt registered [ 3892.989208] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 3894.989079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3895.159403] alg: No test for adler32 (adler32-zlib) [ 3896.536739] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3896.846758] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 3898.607173] Key type lgssc registered [ 3900.200433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.920847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3909.953894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.354984] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3911.381140] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.38/24@tcp [ 3911.399531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.963599] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3920.872975] LustreError: 127305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.905983] LustreError: 127305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3926.006228] LustreError: 127304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.972932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.661541] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3928.735837] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.38/24@tcp [ 3928.819560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.213800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3944.491836] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3953.447446] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.058382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.427026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.621769] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 3977.505811] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3980.858161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3991.031923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.038436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.059191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.080266] Lustre: Skipped 1 previous similar message [ 3996.129442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.137609] Lustre: Skipped 1 previous similar message [ 3996.986790] Lustre: server umount lustre-OST0000 complete [ 4005.347186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4005.352106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.359803] Lustre: Skipped 1 previous similar message [ 4005.363971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.370296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.882689] Lustre: server umount lustre-MDT0000 complete [ 4011.495908] LustreError: 127301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.529030] LustreError: 127301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4011.659276] LustreError: 127285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789020148 with bad export cookie 17592326816619346042 [ 4011.688057] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.294503] Lustre: server umount lustre-MDT0001 complete [ 4024.845586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4028.082794] Key type lgssc unregistered [ 4028.553817] LNet: 130905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4028.562240] LNetError: 130905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4029.606849] LNet: Removed LNI 192.168.202.138@tcp [ 4030.625249] Key type .llcrypt unregistered [ 4030.627680] Key type ._llcrypt unregistered [ 4060.465495] Key type ._llcrypt registered [ 4060.467186] Key type .llcrypt registered [ 4060.529257] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4078.879448] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4079.713049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4079.844188] alg: No test for adler32 (adler32-zlib) [ 4080.966194] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 4081.254676] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4083.063821] Key type lgssc registered [ 4084.569143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4119.632791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4119.648841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.994597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4121.018211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4121.170858] Lustre: lustre-MDT0000: new disk, initializing [ 4121.329708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.364712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4126.510683] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4136.683240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.811546] Lustre: 134304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4136.875622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4136.881368] Lustre: Skipped 1 previous similar message [ 4136.944516] Lustre: lustre-MDT0001: new disk, initializing [ 4137.062362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.148384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4137.164859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4141.390254] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4152.213849] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4158.505406] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4164.981326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.184745] Lustre: lustre-OST0000: new disk, initializing [ 4165.188830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4165.199665] Lustre: 136272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4165.305711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.509615] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4172.411070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4172.424368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4172.621728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4181.148100] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.053732] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4186.312834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4191.973415] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4192.169936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4198.372969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4198.377748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.382254] LustreError: Skipped 1 previous similar message [ 4198.383982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.394673] Lustre: Skipped 1 previous similar message [ 4198.412804] Lustre: Skipped 1 previous similar message [ 4200.488458] Lustre: server umount lustre-OST0000 complete [ 4208.610777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.611968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4208.612708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.629476] Lustre: Skipped 1 previous similar message [ 4211.603446] Lustre: server umount lustre-MDT0000 complete [ 4215.858850] LustreError: 134297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789020352 with bad export cookie 7361027411836267992 [ 4215.862294] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.869045] LustreError: 134297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4216.298389] Lustre: server umount lustre-MDT0001 complete [ 4225.395817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.787519] LustreError: 138490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.803682] LustreError: 138490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4225.849580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.581736] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4231.138223] LustreError: 138491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.283465] LustreError: 138490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.723084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.218666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.844344] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4251.529753] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4257.366300] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4265.238437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4265.565852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.437295] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4280.913784] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4285.540877] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4285.813658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4291.172081] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4291.405970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4301.710563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.942922] Lustre: lustre-OST0001: new disk, initializing [ 4301.948825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4301.953529] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (not set up) [ 4301.957978] Lustre: 141859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4301.974619] Lustre: Skipped 1 previous similar message [ 4302.090170] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4308.158382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4308.181524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4308.380301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4310.299436] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4320.575425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4320.916829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4326.849828] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4327.005596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4334.425828] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95400424b000.ost_server_uuid 50 [ 4336.590606] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95400424b000.ost_server_uuid in IDLE state after 0 sec [ 4347.510422] Lustre: server umount lustre-OST0001 complete [ 4349.421660] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4349.433394] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.449115] LustreError: Skipped 1 previous similar message [ 4349.451891] LustreError: 141860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.500117] Lustre: Skipped 1 previous similar message [ 4352.993451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.010108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.048685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.542547] LustreError: 140311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.613474] LustreError: 140311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4358.832680] Lustre: server umount lustre-OST0000 complete [ 4366.714115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.733611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.739183] Lustre: Skipped 2 previous similar messages [ 4366.787813] Lustre: Skipped 3 previous similar messages [ 4373.342899] Lustre: server umount lustre-MDT0000 complete [ 4375.018793] LustreError: 139218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.053728] LustreError: 139218:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4378.809266] LustreError: 138472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789020515 with bad export cookie 7361027411836268874 [ 4378.815714] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.824967] LustreError: 138472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4379.171532] Lustre: server umount lustre-MDT0001 complete [ 4402.682611] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4406.359292] Key type lgssc unregistered [ 4406.719231] LNet: 144868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4406.736964] LNetError: 144868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4407.781197] LNet: Removed LNI 192.168.202.138@tcp [ 4408.928126] Key type .llcrypt unregistered [ 4408.929720] Key type ._llcrypt unregistered [ 4455.790852] Key type ._llcrypt registered [ 4455.793459] Key type .llcrypt registered [ 4455.897863] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4472.006235] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4473.782527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.859919] alg: No test for adler32 (adler32-zlib) [ 4475.026810] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 4475.277430] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4476.991767] Key type lgssc registered [ 4478.292291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4522.724595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4522.748498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.046360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4524.072866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4524.186494] Lustre: lustre-MDT0000: new disk, initializing [ 4524.246748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.259721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4529.229865] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4540.386875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.548667] Lustre: 149025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4540.624405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4540.631150] Lustre: Skipped 1 previous similar message [ 4540.718764] Lustre: lustre-MDT0001: new disk, initializing [ 4540.813068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.859444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4540.872890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4546.099854] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4556.669105] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4563.265713] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4571.736407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.267435] Lustre: lustre-OST0000: new disk, initializing [ 4572.285263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4572.292578] Lustre: 150996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4572.458294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.849273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4577.862359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4577.978934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4580.087431] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4589.842223] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4594.749707] Lustre: Failing over lustre-OST0000 [ 4594.963802] Lustre: server umount lustre-OST0000 complete [ 4613.087185] Lustre: 147250:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789020733/real 1789020733] req@00000000685a8f0a x1875924076234880/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789020749 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4613.105129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.288183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.816472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.833819] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4614.815397] Lustre: 147248:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789020735/real 1789020735] req@0000000001df71a1 x1875924076236160/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789020751 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4614.933168] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4615.209421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.743490] Lustre: 147250:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789020740/real 1789020740] req@00000000a8b48182 x1875924076237312/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789020756 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4619.759300] Lustre: 147250:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4622.111879] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4623.525925] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4623.532184] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4624.607259] Lustre: 147250:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789020745/real 1789020745] req@00000000dc03431c x1875924076238464/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789020761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.644976] Lustre: 147250:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4626.487632] Lustre: Failing over lustre-MDT0000 [ 4627.181865] Lustre: server umount lustre-MDT0000 complete [ 4627.940875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4627.953682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.570081] LustreError: 149839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.616485] LustreError: 149839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4635.720156] LustreError: 149035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.689439] LustreError: 149039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.723382] LustreError: 149039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4640.841113] LustreError: 149033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.309439] Lustre: 147251:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789020764/real 1789020764] req@000000000f8c3da0 x1875924076249728/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789020780 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4644.354275] Lustre: 147251:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4644.371517] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4645.966663] LustreError: 149035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.004825] LustreError: 149035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4648.445028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.542382] LustreError: 147247:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000e63d6c9 x1875924076254976/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4653.975721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.036674] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4656.219098] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4658.969036] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4659.184377] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4659.196264] Lustre: Skipped 1 previous similar message [ 4659.248486] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4659.277664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4669.420948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.423750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.433716] Lustre: Skipped 3 previous similar messages [ 4674.193676] Lustre: server umount lustre-OST0000 complete [ 4679.650278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.656195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4679.668920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4679.691287] Lustre: Skipped 2 previous similar messages [ 4683.673135] Lustre: server umount lustre-MDT0000 complete [ 4684.769786] LustreError: 149035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.784705] LustreError: 149035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 4687.126057] LustreError: 149020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789020823 with bad export cookie 12692573304315773837 [ 4687.138399] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.607679] Lustre: server umount lustre-MDT0001 complete [ 4699.654951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4702.734417] Key type lgssc unregistered [ 4703.411156] LNet: 154660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4703.423209] LNetError: 154660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4703.451273] LNet: Removed LNI 192.168.202.138@tcp [ 4704.681975] Key type .llcrypt unregistered [ 4704.686490] Key type ._llcrypt unregistered [ 4747.598218] Key type ._llcrypt registered [ 4747.600727] Key type .llcrypt registered [ 4747.675776] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4749.205970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4749.217864] alg: No test for adler32 (adler32-zlib) [ 4751.368467] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 4751.948907] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4753.695559] Key type lgssc registered [ 4755.116898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4764.640336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4764.691287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.403272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.784174] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4775.905697] LustreError: 156179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.921120] LustreError: 156179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4781.030462] LustreError: 156178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.854391] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4786.145769] LustreError: 156179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.266098] LustreError: 156207:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4786.279350] LustreError: 156207:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4786.290488] LustreError: 156207:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4786.613328] Lustre: server umount lustre-MDT0000 complete [ 4803.540737] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4812.931379] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4850.486695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.776451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4850.832605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4850.985387] Lustre: lustre-MDT0000: new disk, initializing [ 4851.109202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.135017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4855.820593] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4867.091327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.279974] Lustre: 159513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.366124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4867.371152] Lustre: Skipped 1 previous similar message [ 4867.493591] Lustre: lustre-MDT0001: new disk, initializing [ 4867.638214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4867.678430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4867.692700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4872.529374] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4885.369519] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4892.343989] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4900.889987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4901.317523] Lustre: lustre-OST0000: new disk, initializing [ 4901.328517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4901.335974] Lustre: 161480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4901.488195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.123985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4907.140079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4907.334967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4909.403985] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4921.642511] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.878843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4928.046841] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:17:43 (1789021063) [ 5224.935882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.938230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.963818] Lustre: Skipped 2 previous similar messages [ 5224.979904] Lustre: Skipped 2 previous similar messages [ 5230.051695] LustreError: 159517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.089546] LustreError: 159517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5230.779770] Lustre: server umount lustre-MDT0000 complete [ 5235.179464] LustreError: 159522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.231419] LustreError: 159522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5235.771919] LustreError: 160305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789021372 with bad export cookie 783764431687187576 [ 5235.779385] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5235.788937] LustreError: 160305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.289184] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.342726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5242.669806] Lustre: server umount lustre-MDT0001 complete [ 5261.279327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5261.596218] Lustre: server umount lustre-OST0000 complete [ 5272.623163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5285.334549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5336.255468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.854039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5336.946464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5337.138639] Lustre: lustre-MDT0000: new disk, initializing [ 5337.350948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.430684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5344.151476] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5357.761055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.900218] Lustre: 166158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5357.941688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5357.944888] Lustre: Skipped 1 previous similar message [ 5357.998166] Lustre: lustre-MDT0001: new disk, initializing [ 5358.076741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5358.141572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5358.171887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5363.712580] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5374.334895] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5381.879478] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5388.623855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5388.809132] Lustre: lustre-OST0000: new disk, initializing [ 5388.814896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5388.829766] Lustre: 168124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5389.034174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.730207] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5398.066362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5398.072949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5398.219299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5406.650623] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5411.870482] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5412.144133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5418.009623] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5418.295602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5423.586314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5423.609830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.639315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5424.097679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.027375] Lustre: server umount lustre-OST0000 complete [ 5434.336394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.338910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5434.355757] Lustre: Skipped 2 previous similar messages [ 5434.370584] Lustre: Skipped 3 previous similar messages [ 5438.026972] Lustre: server umount lustre-MDT0000 complete [ 5439.466637] LustreError: 166170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.505500] LustreError: 166170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5443.713379] LustreError: 166149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789021580 with bad export cookie 783764431687346406 [ 5443.724927] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5443.730558] LustreError: 166149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5444.220892] Lustre: server umount lustre-MDT0001 complete [ 5461.181725] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:26:36 (1789021596) [ 5482.628795] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5493.109957] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5538.572244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.127704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5539.173724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5539.408183] Lustre: lustre-MDT0000: new disk, initializing [ 5539.611626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.634447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5546.149643] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5559.344071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.487930] Lustre: 172885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5559.762398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5567.326888] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5581.367797] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5587.349888] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5594.427447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5595.135777] Lustre: lustre-OST0000: new disk, initializing [ 5595.147026] Lustre: Skipped 1 previous similar message [ 5595.149177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5595.158513] Lustre: Skipped 2 previous similar messages [ 5595.167801] Lustre: 174863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5600.901878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5600.911027] Lustre: Skipped 1 previous similar message [ 5600.921237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5601.090268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5602.397650] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5612.578867] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.728271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5621.729360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.740338] LustreError: Skipped 1 previous similar message [ 5621.740996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5621.777687] Lustre: Skipped 1 previous similar message [ 5626.488972] Lustre: server umount lustre-OST0000 complete [ 5631.457106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5631.482788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.503361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5631.516482] Lustre: Skipped 1 previous similar message [ 5645.279380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5646.124409] Lustre: server umount lustre-MDT0000 complete [ 5647.348040] LustreError: 172891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.421159] LustreError: 172891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5653.836086] LustreError: 174862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789021790 with bad export cookie 783764431687347288 [ 5653.837434] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.847330] LustreError: 174862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5654.290211] Lustre: server umount lustre-MDT0001 complete [ 5665.049636] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:30:00 (1789021800) [ 5683.988425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5699.701355] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5739.457314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.675526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.706976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.800837] Lustre: lustre-MDT0000: new disk, initializing [ 5739.932746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.945565] Lustre: Skipped 2 previous similar messages [ 5739.965510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5746.378567] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5758.095890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.298650] Lustre: 179209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5758.853994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5764.170365] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5778.436142] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5790.862769] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5798.565382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.069511] Lustre: 181191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5806.162816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5806.304734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5807.387588] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5818.140640] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5827.041625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.045345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.080426] Lustre: Skipped 2 previous similar messages [ 5827.088680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.107555] Lustre: Skipped 6 previous similar messages [ 5831.843697] Lustre: server umount lustre-OST0000 complete [ 5842.977841] Lustre: server umount lustre-MDT0000 complete [ 5847.520681] LustreError: 179217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.538531] LustreError: 179217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5848.020988] LustreError: 179202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789021984 with bad export cookie 783764431687348373 [ 5848.032257] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5848.774249] Lustre: server umount lustre-MDT0001 complete [ 5860.988411] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5863.986478] Key type lgssc unregistered [ 5864.289178] LNet: 183045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5864.299022] LNetError: 183045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5864.318749] LNet: Removed LNI 192.168.202.138@tcp [ 5865.291839] Key type .llcrypt unregistered [ 5865.297677] Key type ._llcrypt unregistered [ 5893.955969] Key type ._llcrypt registered [ 5893.960555] Key type .llcrypt registered [ 5894.138775] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5896.235397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.295154] alg: No test for adler32 (adler32-zlib) [ 5897.446334] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 5897.823635] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5899.648492] Key type lgssc registered [ 5901.214192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5912.174666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5912.190255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.738858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.705850] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5924.335235] LustreError: 184464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.361844] LustreError: 184464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5929.445490] LustreError: 184465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.912127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.437936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5935.439439] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5945.442131] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5953.372213] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5963.720829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.303233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.351638] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5979.789981] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5983.744663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6005.221938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6005.234465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.253628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.323696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.332860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.333268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6011.519701] Lustre: server umount lustre-OST0000 complete [ 6017.505635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6017.510638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.524070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6017.526775] Lustre: Skipped 1 previous similar message [ 6019.553651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.556216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.598391] Lustre: Skipped 1 previous similar message [ 6024.549798] Lustre: server umount lustre-MDT0000 complete [ 6024.677759] LustreError: 184464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.692967] LustreError: 184464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6029.010375] LustreError: 184446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789022165 with bad export cookie 6565000618931691347 [ 6029.013951] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.017106] LustreError: 184446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6029.269294] Lustre: server umount lustre-MDT0001 complete [ 6041.723460] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6044.800412] Key type lgssc unregistered [ 6045.044748] LNet: 188022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6045.050522] LNetError: 188022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6046.117979] LNet: Removed LNI 192.168.202.138@tcp [ 6047.257065] Key type .llcrypt unregistered [ 6047.259762] Key type ._llcrypt unregistered [ 6074.852865] Key type ._llcrypt registered [ 6074.855467] Key type .llcrypt registered [ 6074.964604] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6076.704067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6076.800953] alg: No test for adler32 (adler32-zlib) [ 6078.010327] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6078.287606] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6080.023437] Key type lgssc registered [ 6081.452748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6091.542504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6091.576488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.142390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.985741] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6103.522853] LustreError: 189437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.544977] LustreError: 189437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6108.644289] LustreError: 189438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.842755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.433295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6115.437317] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6123.109463] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6129.474437] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.302650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6137.725246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.758481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6144.964599] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6155.223424] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6162.065996] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 6164.453082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.487728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.537640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6165.474210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6165.500313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.118400] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 6167.140435] Lustre: Skipped 2 previous similar messages [ 6170.615807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6175.718970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6175.750408] Lustre: Skipped 3 previous similar messages [ 6176.223205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6176.746504] Lustre: server umount lustre-OST0000 complete [ 6188.946715] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9540065c4800.ost_server_uuid 50 [ 6191.950488] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6211.554190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.570262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6211.573986] Lustre: Skipped 1 previous similar message [ 6211.587322] Lustre: Skipped 1 previous similar message [ 6215.283639] Lustre: server umount lustre-MDT0000 complete [ 6216.673253] LustreError: 189437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.682883] LustreError: 189437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6219.636862] LustreError: 189419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789022356 with bad export cookie 7429479937401979701 [ 6219.640866] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6219.651099] LustreError: 189419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6219.948712] Lustre: server umount lustre-MDT0001 complete [ 6229.580666] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:39:24 (1789022364) [ 6237.181780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.771864] LustreError: 193356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.875271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.306769] LustreError: 193357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.401133] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6253.103285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.540201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6253.938237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.888806] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6271.325597] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6278.028910] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.349868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.732985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.447290] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6299.678908] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6307.311223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6307.567239] Lustre: lustre-OST0001: new disk, initializing [ 6307.572197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6307.578095] Lustre: 196199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6307.727396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6308.803326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6313.612428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6313.621951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6313.768255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6315.240061] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6327.431091] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.554208] Lustre: lustre-OST0000: Not available for connect from 192.168.202.38@tcp (stopping) [ 6334.300832] Lustre: server umount lustre-OST0000 complete [ 6334.442350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6334.466882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.502989] LustreError: 195181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.522585] LustreError: 195181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6342.895607] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6343.257777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6348.708844] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6349.114531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6353.999653] LustreError: 195181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.030968] LustreError: 195181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6368.508832] Lustre: server umount lustre-OST0001 complete [ 6375.393256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.417917] Lustre: Skipped 1 previous similar message [ 6383.763497] Lustre: server umount lustre-MDT0000 complete [ 6385.567330] Lustre: 188747:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789022506/real 1789022506] req@00000000afd59ca2 x1875925757172480/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789022522 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6385.606351] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.627868] Lustre: Skipped 1 previous similar message [ 6389.431535] LustreError: 194093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789022525 with bad export cookie 7429479937401980772 [ 6389.435218] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6389.438194] LustreError: 194093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6389.801101] Lustre: server umount lustre-MDT0001 complete [ 6416.651764] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:42:31 (1789022551) [ 6423.763096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.082853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6424.118525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6424.320779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.367323] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6438.434939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.530938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6438.567683] Lustre: 199642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6444.357708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6452.005313] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6458.001690] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6466.817458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6466.962935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6474.573292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6477.692858] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6489.599840] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6501.675739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.747292] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6502.077787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.090314] Lustre: Skipped 2 previous similar messages [ 6510.793947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6511.295327] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6521.544278] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.920570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6525.928824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.939475] LustreError: Skipped 1 previous similar message [ 6525.943122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.943131] Lustre: Skipped 4 previous similar messages [ 6525.996461] Lustre: Skipped 1 previous similar message [ 6530.571673] Lustre: server umount lustre-OST0000 complete [ 6531.573084] LustreError: 203023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.608963] LustreError: 203023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6551.526254] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.750315] Lustre: server umount lustre-OST0001 complete [ 6561.760813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6561.764247] LustreError: Skipped 1 previous similar message [ 6561.770865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.782299] Lustre: Skipped 2 previous similar messages [ 6564.415961] Lustre: server umount lustre-MDT0000 complete [ 6568.437154] LustreError: 199634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789022704 with bad export cookie 7429479937401982179 [ 6568.440848] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.446919] LustreError: 199634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6568.937318] Lustre: server umount lustre-MDT0001 complete [ 6595.628671] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:45:30 (1789022730) [ 6613.770342] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6626.557754] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6670.547626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.886928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6670.908643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6670.979236] Lustre: lustre-MDT0000: new disk, initializing [ 6671.037552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.054124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6676.078196] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6687.123815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.232107] Lustre: 207740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6687.316444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6687.330790] Lustre: Skipped 1 previous similar message [ 6687.443756] Lustre: lustre-MDT0001: new disk, initializing [ 6687.618811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6687.634761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6692.815844] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6703.643859] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6710.802565] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6720.334347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6720.704899] Lustre: lustre-OST0000: new disk, initializing [ 6720.712151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6720.723225] Lustre: 209709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6728.787556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6728.810496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6728.981029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6729.373084] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6741.832475] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.314378] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6748.684861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6755.134912] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6755.413526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6759.909895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6759.918307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.918994] LustreError: Skipped 1 previous similar message [ 6759.924131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6759.929867] Lustre: Skipped 2 previous similar messages [ 6759.962154] Lustre: Skipped 11 previous similar messages [ 6763.620844] Lustre: server umount lustre-OST0000 complete [ 6774.652319] Lustre: server umount lustre-MDT0000 complete [ 6775.265091] LustreError: 207745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.297852] LustreError: 207745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6779.101238] LustreError: 207731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789022915 with bad export cookie 7429479937401983964 [ 6779.106913] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.129663] LustreError: 207731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6779.892861] Lustre: server umount lustre-MDT0001 complete [ 6790.230299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.207677] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6804.501213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.103070] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6818.702616] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6824.738617] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6831.943984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6838.301425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6847.063200] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.734925] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6852.942462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6857.867163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6858.112871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6860.260987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.273592] LustreError: Skipped 1 previous similar message [ 6866.588925] Lustre: server umount lustre-OST0000 complete [ 6873.716580] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6878.961882] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6909.287767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.768633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6915.799852] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6915.815428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6915.850839] Lustre: Skipped 1 previous similar message [ 6916.888233] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6926.213791] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6928.742921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6936.547033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6936.549961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.568587] Lustre: Skipped 5 previous similar messages [ 6939.856405] Lustre: server umount lustre-OST0000 complete [ 6950.424378] Lustre: server umount lustre-MDT0000 complete [ 6951.906821] LustreError: 211928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.918293] LustreError: 211928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6954.149585] LustreError: 211913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789023090 with bad export cookie 7429479937401984846 [ 6954.154336] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6954.162578] LustreError: 211913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6954.456348] Lustre: server umount lustre-MDT0001 complete [ 6964.706488] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:51:39 (1789023099) [ 6971.513322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.882234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.887564] Lustre: Skipped 6 previous similar messages [ 6977.628554] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6987.123310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.380493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7001.053172] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7007.034870] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7014.145765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.352967] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7029.782800] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.455893] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7035.673441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7040.339762] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7040.544418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7047.482541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.766783] Lustre: lustre-OST0001: new disk, initializing [ 7047.773851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7047.781421] Lustre: 221271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7053.991940] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7058.020044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7058.029205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7058.174723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7063.239377] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.235147] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7068.376583] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7074.292856] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7074.629361] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7078.881775] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7078.904293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7078.917045] Lustre: Skipped 11 previous similar messages [ 7083.245725] Lustre: server umount lustre-OST0001 complete [ 7090.253336] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7090.510431] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7095.699951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7096.027921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7124.885428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.603210] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7130.624316] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7130.628052] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7130.659552] Lustre: Skipped 1 previous similar message [ 7132.404810] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7140.209794] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.046553] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7158.239138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7158.425528] Lustre: server umount lustre-OST0001 complete [ 7182.171901] Lustre: server umount lustre-OST0000 complete [ 7192.759854] Lustre: server umount lustre-MDT0000 complete [ 7197.603098] LustreError: 217876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789023334 with bad export cookie 7429479937401985812 [ 7197.613164] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.618843] LustreError: 217876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7204.178031] Lustre: server umount lustre-MDT0001 complete [ 7235.322028] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:56:10 (1789023370) [ 7242.898552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.155733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7243.248461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7248.412901] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7257.452514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.560686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7257.622050] Lustre: 226516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7262.452439] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7271.309275] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7278.099064] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7286.281479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.495690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7293.865760] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7303.639314] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.492530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7315.601231] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7323.259614] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7334.130787] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7335.846721] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7342.440475] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7342.741565] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7348.537788] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7348.913828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7355.051958] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff954006c7b000.ost_server_uuid 50 [ 7357.124928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff954006c7b000.ost_server_uuid in IDLE state after 0 sec [ 7358.142871] Lustre: Permanently deactivating lustre-OST0001 [ 7358.159502] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7361.447608] Lustre: Permanently reactivating lustre-OST0001 [ 7367.147084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7367.156425] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.159254] LustreError: Skipped 5 previous similar messages [ 7367.195147] Lustre: Skipped 11 previous similar messages [ 7372.015238] Lustre: server umount lustre-OST0001 complete [ 7372.282193] LustreError: 228361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.307928] LustreError: 228361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 7391.199094] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7391.358277] Lustre: server umount lustre-OST0000 complete [ 7400.735318] Lustre: server umount lustre-MDT0000 complete [ 7404.520270] LustreError: 226507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789023541 with bad export cookie 7429479937401987100 [ 7404.534237] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7404.916595] Lustre: server umount lustre-MDT0001 complete [ 7429.286389] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:59:24 (1789023564) [ 7437.517758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.012294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7438.068723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7438.076947] Lustre: Skipped 5 previous similar messages [ 7443.611215] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7455.091417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.301840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7455.390382] Lustre: 233213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7460.873735] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7468.634260] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7475.827746] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7483.057726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7483.265038] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7483.312444] Lustre: Permanently deactivating lustre-OST0000 [ 7483.329775] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7489.555337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7490.302573] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7497.321954] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7504.446746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.506688] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7504.808522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7504.823060] Lustre: Skipped 11 previous similar messages [ 7511.504481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7514.264939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7519.465806] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7521.401759] Lustre: Permanently reactivating lustre-OST0000 [ 7521.414718] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7521.422510] Lustre: Skipped 2 previous similar messages [ 7521.778352] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7521.779182] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000e1956cb5, cur 1789023658 deadline 1789023656 last 1789023626 [ 7521.784115] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7521.786542] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7521.872663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7548.383518] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7548.487213] Lustre: server umount lustre-OST0001 complete [ 7558.601438] Lustre: server umount lustre-OST0000 complete [ 7570.768220] Lustre: server umount lustre-MDT0000 complete [ 7575.498184] LustreError: 233205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789023711 with bad export cookie 7429479937401988745 [ 7575.509669] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.517046] LustreError: 233205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7576.060597] Lustre: server umount lustre-MDT0001 complete [ 7588.778484] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7591.748827] Key type lgssc unregistered [ 7592.141942] LNet: 238105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7592.160392] LNetError: 238105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7592.193280] LNet: Removed LNI 192.168.202.138@tcp [ 7593.174832] Key type .llcrypt unregistered [ 7593.180080] Key type ._llcrypt unregistered [ 7616.279193] Key type ._llcrypt registered [ 7616.281816] Key type .llcrypt registered [ 7616.371162] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7617.591973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7617.603375] alg: No test for adler32 (adler32-zlib) [ 7618.604873] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7618.811869] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7620.455752] Key type lgssc registered [ 7621.564175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7630.888693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7630.902281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.503347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7637.190733] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7642.092123] LustreError: 239572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.106032] LustreError: 239572:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7646.061452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.170458] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7646.188932] Lustre: 239561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7646.200159] Lustre: Permanently deactivating lustre-MDT0001 [ 7646.210596] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7646.471397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.561136] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7655.941836] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7658.971414] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7664.876629] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7672.774464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7673.306542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.418784] LustreError: 241415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7677.449765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7680.083811] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7682.543216] LustreError: 241412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.653031] LustreError: 241415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.632687] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7690.736514] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7695.864465] LustreError: 241634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.922994] LustreError: 241634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7696.770391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.121223] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.531289] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7702.544491] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7705.716766] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7713.627410] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7716.888756] Lustre: Permanently reactivating lustre-MDT0001 [ 7716.907973] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7716.919365] Lustre: Skipped 2 previous similar messages [ 7722.086967] Lustre: lustre-MDT0001: Client d0f0383e-48c3-45f8-8153-49011e707d70 (at 192.168.202.38@tcp) reconnecting [ 7725.197291] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.221327] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7725.234107] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7725.254724] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7727.856164] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7729.302400] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7735.792129] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7736.084357] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7741.206266] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7741.425575] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7744.109769] Lustre: Permanently deactivating lustre-MDT0001 [ 7744.143243] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7744.151113] Lustre: Skipped 2 previous similar messages [ 7751.796621] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7761.890618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.895456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.911099] Lustre: Skipped 2 previous similar messages [ 7761.928887] Lustre: Skipped 3 previous similar messages [ 7767.012871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.024644] Lustre: Skipped 3 previous similar messages [ 7767.277581] Lustre: server umount lustre-MDT0000 complete [ 7771.125477] LustreError: 239553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789023907 with bad export cookie 1773989001351296705 [ 7771.126960] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.133579] LustreError: 239553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7771.398865] Lustre: server umount lustre-MDT0001 complete [ 7777.544715] Lustre: server umount lustre-OST0000 complete [ 7788.402440] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:05:23 (1789023923) [ 7799.310496] Lustre: server umount lustre-OST0001 complete [ 7806.073816] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7816.021458] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7850.462321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.694449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7850.702947] Lustre: Skipped 2 previous similar messages [ 7850.738450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7850.887945] Lustre: lustre-MDT0000: new disk, initializing [ 7851.012948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.037880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7855.836972] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7866.790475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.922886] Lustre: 247853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7867.013244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7867.019931] Lustre: Skipped 1 previous similar message [ 7867.137119] Lustre: lustre-MDT0001: new disk, initializing [ 7867.280307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7867.323369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7867.337164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7872.604349] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7882.496788] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7889.159889] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7897.754040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.277534] Lustre: lustre-OST0000: new disk, initializing [ 7898.285726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7898.303653] Lustre: 249821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.508150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.330794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7904.339877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7904.483879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7906.463979] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7916.242231] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7918.949708] LustreError: 247861:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7928.508705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.729188] Lustre: lustre-OST0001: new disk, initializing [ 7928.732539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7928.737792] Lustre: 250936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7928.991769] LustreError: 247861:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7929.025745] LustreError: 247860:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7935.253812] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7936.142100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7936.159120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7936.238808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7939.049070] LustreError: 247860:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7945.616479] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7947.041537] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7950.449575] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 7951.865796] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.871193] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7951.904815] Lustre: Skipped 2 previous similar messages [ 7955.543467] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 7955.560563] Lustre: Skipped 2 previous similar messages [ 7960.651786] Lustre: lustre-OST0001: Not available for connect from 192.168.202.38@tcp (stopping) [ 7960.676870] Lustre: Skipped 2 previous similar messages [ 7963.104805] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7963.292711] Lustre: server umount lustre-OST0001 complete [ 7965.768258] LustreError: 251327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.791035] LustreError: 251327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7981.537966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7981.540929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.550514] LustreError: Skipped 2 previous similar messages [ 7981.553681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7981.569679] Lustre: Skipped 1 previous similar message [ 7981.608338] Lustre: Skipped 3 previous similar messages [ 7982.564621] LustreError: 249807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.598977] LustreError: 249807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7987.466842] Lustre: server umount lustre-OST0000 complete [ 7992.801547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.920730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.934123] Lustre: Skipped 5 previous similar messages [ 7998.576793] Lustre: server umount lustre-MDT0000 complete [ 8002.223236] LustreError: 248648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024138 with bad export cookie 1773989001351299519 [ 8002.227485] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.232760] LustreError: 248648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8008.692944] Lustre: server umount lustre-MDT0001 complete [ 8022.229991] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8026.305519] Key type lgssc unregistered [ 8026.569564] LNet: 252974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8026.583235] LNetError: 252974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8026.596811] LNet: Removed LNI 192.168.202.138@tcp [ 8027.511842] Key type .llcrypt unregistered [ 8027.516550] Key type ._llcrypt unregistered [ 8073.372361] Key type ._llcrypt registered [ 8073.374593] Key type .llcrypt registered [ 8073.516783] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8075.269673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8075.366835] alg: No test for adler32 (adler32-zlib) [ 8076.550695] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8076.874781] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8078.672144] Key type lgssc registered [ 8079.629861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8088.034657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8088.053244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.290387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8089.321964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8089.449434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.170679] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8102.942069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.002103] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8103.016374] Lustre: 255193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8103.287283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8107.699799] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8115.446207] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8121.278544] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8130.579756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8130.735727] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8131.038748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.447805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8138.578265] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8145.682631] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8199.143700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8199.146604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.147768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.199850] LustreError: Skipped 1 previous similar message [ 8204.220946] Lustre: server umount lustre-OST0000 complete [ 8209.090650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8223.057220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8223.538158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.831469] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8238.901598] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8238.939704] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8238.940664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8238.971728] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8285.157933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.160487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.171822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.171829] Lustre: Skipped 1 previous similar message [ 8285.186535] Lustre: Skipped 2 previous similar messages [ 8286.496619] Lustre: server umount lustre-OST0000 complete [ 8290.272354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.285382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.292569] Lustre: Skipped 2 previous similar messages [ 8295.393475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8295.399343] Lustre: Skipped 1 previous similar message [ 8296.630399] Lustre: server umount lustre-MDT0000 complete [ 8300.651267] LustreError: 255184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024437 with bad export cookie 5967523733617188444 [ 8300.657853] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8300.671263] LustreError: 255184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8301.043883] Lustre: server umount lustre-MDT0001 complete [ 8313.613234] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8316.919547] Key type lgssc unregistered [ 8317.236611] LNet: 260611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8317.243878] LNetError: 260611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8317.267741] LNet: Removed LNI 192.168.202.138@tcp [ 8318.403455] Key type .llcrypt unregistered [ 8318.416354] Key type ._llcrypt unregistered [ 8343.455379] Key type ._llcrypt registered [ 8343.458720] Key type .llcrypt registered [ 8343.554280] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8344.807171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8344.832713] alg: No test for adler32 (adler32-zlib) [ 8345.987200] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8346.357338] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8348.039392] Key type lgssc registered [ 8349.284736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8357.999069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8358.026690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.446050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.998527] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8368.623101] LustreError: 262026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.642387] LustreError: 262026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8372.703838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.209773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8378.122976] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8386.122582] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.491987] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.695548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.255257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.352055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8408.783600] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8417.724682] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8443.368648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8443.380246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.388253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8448.489856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8448.507835] Lustre: Skipped 2 previous similar messages [ 8449.019229] Lustre: server umount lustre-OST0000 complete [ 8453.601248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.618995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8453.619390] Lustre: Skipped 2 previous similar messages [ 8453.638097] Lustre: Skipped 1 previous similar message [ 8458.722445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.738699] Lustre: Skipped 1 previous similar message [ 8459.876487] Lustre: server umount lustre-MDT0000 complete [ 8463.511828] LustreError: 262008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024600 with bad export cookie 5641841977251335506 [ 8463.514746] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8463.521807] LustreError: 262008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8463.876852] Lustre: server umount lustre-MDT0001 complete [ 8476.853293] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8480.164815] Key type lgssc unregistered [ 8480.521360] LNet: 266332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.532626] LNetError: 266332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8481.576211] LNet: Removed LNI 192.168.202.138@tcp [ 8482.593295] Key type .llcrypt unregistered [ 8482.597101] Key type ._llcrypt unregistered [ 8503.873667] Key type ._llcrypt registered [ 8503.875767] Key type .llcrypt registered [ 8503.981378] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8505.374484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8505.531842] alg: No test for adler32 (adler32-zlib) [ 8506.783909] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8507.160726] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8508.919345] Key type lgssc registered [ 8510.363114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8519.524643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8519.567334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.013866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.361789] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8531.427868] LustreError: 267551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.447888] LustreError: 267551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8536.132953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8536.641317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8542.085333] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8551.194615] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8557.371878] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8564.379285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8564.823240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.883506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8571.741825] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8579.989456] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8596.960205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8596.965227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.980917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8597.990493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8598.003536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8598.028787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.514402] Lustre: server umount lustre-OST0000 complete [ 8608.227838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8608.231372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.247726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8608.263788] Lustre: Skipped 1 previous similar message [ 8608.272890] Lustre: Skipped 2 previous similar messages [ 8612.313099] Lustre: server umount lustre-MDT0000 complete [ 8613.345592] LustreError: 269686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.368465] LustreError: 269686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8616.459962] LustreError: 267532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024752 with bad export cookie 13950327820044124723 [ 8616.470754] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.478076] LustreError: 267532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8617.124852] Lustre: server umount lustre-MDT0001 complete [ 8630.047971] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8633.189894] Key type lgssc unregistered [ 8633.469687] LNet: 271230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8633.477222] LNetError: 271230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8634.533787] LNet: Removed LNI 192.168.202.138@tcp [ 8635.465202] Key type .llcrypt unregistered [ 8635.469049] Key type ._llcrypt unregistered [ 8661.832807] Key type ._llcrypt registered [ 8661.835310] Key type .llcrypt registered [ 8661.994163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8663.170193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8663.229638] alg: No test for adler32 (adler32-zlib) [ 8664.656199] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8664.976224] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8666.679223] Key type lgssc registered [ 8668.166562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8676.884444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8676.916594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.527938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.492881] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8689.138326] LustreError: 272644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8689.167971] LustreError: 272644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8690.751240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8691.165457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8695.826341] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8704.381025] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8710.489698] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8718.877980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8719.241147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8725.427982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8727.130747] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8735.437805] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8766.437671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8766.441982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8766.443591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8766.452160] Lustre: Skipped 1 previous similar message [ 8766.458777] Lustre: Skipped 1 previous similar message [ 8767.652491] Lustre: server umount lustre-OST0000 complete [ 8773.088738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8773.102318] LustreError: Skipped 1 previous similar message [ 8773.113898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.137292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.672619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.675862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.696392] Lustre: Skipped 1 previous similar message [ 8781.794762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.806052] Lustre: Skipped 1 previous similar message [ 8786.399384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8786.590264] Lustre: server umount lustre-MDT0000 complete [ 8786.914078] LustreError: 272638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.932462] LustreError: 272638:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8790.860246] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024927 with bad export cookie 12196623320523336824 [ 8790.879023] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.883825] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8791.210969] Lustre: server umount lustre-MDT0001 complete [ 8802.476415] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8806.198849] Key type lgssc unregistered [ 8806.510929] LNet: 276950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8806.515448] LNetError: 276950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8806.537494] LNet: Removed LNI 192.168.202.138@tcp [ 8807.357681] Key type .llcrypt unregistered [ 8807.362217] Key type ._llcrypt unregistered [ 8827.709837] Key type ._llcrypt registered [ 8827.711634] Key type .llcrypt registered [ 8827.858845] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 8828.901420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8829.000822] alg: No test for adler32 (adler32-zlib) [ 8830.176046] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8830.429771] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 8832.168797] Key type lgssc registered [ 8833.631876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8842.172546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8842.204432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.884390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.581430] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8854.499160] LustreError: 278183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.512514] LustreError: 278183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8857.777952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.326827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8863.935326] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8872.784659] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8878.767145] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8885.320443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8885.635957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.795889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8892.281822] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 8901.360490] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8915.425529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8915.438882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.455124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8918.521816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.523609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8923.618440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8923.638982] Lustre: Skipped 2 previous similar messages [ 8928.745287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8928.769027] Lustre: Skipped 1 previous similar message [ 8929.247161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8929.578542] Lustre: server umount lustre-OST0000 complete [ 8933.856646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.860570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.882911] Lustre: Skipped 1 previous similar message [ 8933.902887] Lustre: Skipped 1 previous similar message [ 8940.095550] Lustre: server umount lustre-MDT0000 complete [ 8944.084762] LustreError: 278151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025080 with bad export cookie 17355401173826487111 [ 8944.104238] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8944.107093] LustreError: 278151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8944.970966] Lustre: server umount lustre-MDT0001 complete [ 8956.712297] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 8959.642739] Key type lgssc unregistered [ 8960.031681] LNet: 281884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8960.038555] LNetError: 281884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8960.056172] LNet: Removed LNI 192.168.202.138@tcp [ 8961.000728] Key type .llcrypt unregistered [ 8961.006374] Key type ._llcrypt unregistered [ 8973.695338] Key type ._llcrypt registered [ 8973.697594] Key type .llcrypt registered [ 8973.797127] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8992.044044] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 9010.192826] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9011.941224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9012.143978] alg: No test for adler32 (adler32-zlib) [ 9013.321557] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 9013.737986] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9015.487221] Key type lgssc registered [ 9016.844776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9061.436181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9061.481513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.732179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9062.757472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9062.846444] Lustre: lustre-MDT0000: new disk, initializing [ 9062.932141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9062.948240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9068.076336] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9079.803800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.932735] Lustre: 285436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9079.998859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9080.002411] Lustre: Skipped 1 previous similar message [ 9080.089414] Lustre: lustre-MDT0001: new disk, initializing [ 9080.183084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9080.238324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9080.255356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9084.789240] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9096.171205] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9102.709397] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9110.964040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9111.308292] Lustre: lustre-OST0000: new disk, initializing [ 9111.317049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9111.334758] Lustre: 287404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9111.472685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9118.918779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9121.347609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9121.362704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9121.454305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9129.194646] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9133.870731] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9134.140285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9139.174848] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9139.410362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9142.243719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9142.245446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.253213] LustreError: Skipped 1 previous similar message [ 9142.255096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9142.305859] Lustre: Skipped 1 previous similar message [ 9147.426444] Lustre: server umount lustre-OST0000 complete [ 9151.971961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9151.979556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.997458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9152.003934] Lustre: Skipped 1 previous similar message [ 9157.619870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9157.627448] Lustre: Skipped 3 previous similar messages [ 9157.967215] Lustre: server umount lustre-MDT0000 complete [ 9161.520029] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025298 with bad export cookie 17552829973233725163 [ 9161.534903] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9161.542605] LustreError: 285427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9161.852590] Lustre: server umount lustre-MDT0001 complete [ 9171.138619] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:28:26 (1789025306) [ 9178.301682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.862454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.966639] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9184.234138] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.267535] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9189.350733] LustreError: 289817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.255971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.464707] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.918542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9200.478580] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9209.902563] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9218.856975] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9228.028733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9228.413500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.562866] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9244.416958] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9261.541394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9261.546755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9261.559797] Lustre: Skipped 1 previous similar message [ 9261.565544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9264.612388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9264.638643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.318720] Lustre: server umount lustre-OST0000 complete [ 9274.848510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.853861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9274.871219] Lustre: Skipped 1 previous similar message [ 9274.890051] Lustre: Skipped 3 previous similar messages [ 9279.490698] Lustre: server umount lustre-MDT0000 complete [ 9279.977459] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9279.999875] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9284.414672] LustreError: 291661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025420 with bad export cookie 17552829973233726052 [ 9284.420247] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9284.433128] LustreError: 291661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9285.165648] Lustre: server umount lustre-MDT0001 complete [ 9297.369418] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9300.236721] Key type lgssc unregistered [ 9300.447762] LNet: 293375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9300.450782] LNetError: 293375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9300.462700] LNet: Removed LNI 192.168.202.138@tcp [ 9301.246756] Key type .llcrypt unregistered [ 9301.249331] Key type ._llcrypt unregistered [ 9334.048119] Key type ._llcrypt registered [ 9334.055699] Key type .llcrypt registered [ 9334.202615] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 9350.764841] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9351.889705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9352.099020] alg: No test for adler32 (adler32-zlib) [ 9353.135543] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 9353.477469] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9355.144860] Key type lgssc registered [ 9356.532040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9402.608352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9402.679717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.007438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9404.026137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9404.092641] Lustre: lustre-MDT0000: new disk, initializing [ 9404.193983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.224372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9409.475315] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9420.760489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.873100] Lustre: 297200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9420.923180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9420.927946] Lustre: Skipped 1 previous similar message [ 9420.986331] Lustre: lustre-MDT0001: new disk, initializing [ 9421.074684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9421.121103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9421.146132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9426.431481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9437.397835] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9442.998733] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9450.414211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9450.750455] Lustre: lustre-OST2710: new disk, initializing [ 9450.755723] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9450.764954] Lustre: 299169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9450.884434] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9456.227177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9456.240179] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9456.340190] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9457.689164] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9468.285417] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9476.401577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9476.606624] Lustre: lustre-OST03e8: new disk, initializing [ 9476.610803] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9476.621815] Lustre: 300233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9476.712351] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9482.882110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9482.896290] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9483.041581] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9483.093680] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9490.983347] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.666279] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9500.018316] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9506.523661] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9506.844776] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9512.830889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9513.123134] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9518.520449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9518.700572] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9524.206308] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.209962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9524.227950] Lustre: Skipped 2 previous similar messages [ 9524.247393] Lustre: Skipped 3 previous similar messages [ 9526.318081] Lustre: server umount lustre-MDT0000 complete [ 9530.458742] LustreError: 299168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025666 with bad export cookie 12471861327914152952 [ 9530.464631] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.466937] LustreError: 299168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9530.799815] Lustre: server umount lustre-MDT0001 complete [ 9545.456598] Lustre: server umount lustre-OST2710 complete [ 9559.457916] Lustre: server umount lustre-OST03e8 complete [ 9582.171911] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 9593.242669] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9630.729515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9631.071814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9631.111080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9631.233392] Lustre: lustre-MDT0000: new disk, initializing [ 9631.306694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9631.321743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9636.626929] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9647.753113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.932386] Lustre: 305490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9648.084365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9648.088138] Lustre: Skipped 1 previous similar message [ 9648.197242] Lustre: lustre-MDT0001: new disk, initializing [ 9648.330957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9648.402048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9648.426299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9654.352965] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9664.368470] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9670.409842] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9676.971688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9677.309101] Lustre: lustre-OST0000: new disk, initializing [ 9677.315151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9677.325471] Lustre: 307457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9677.432941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.776122] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9684.593657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9684.599415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9684.763287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9696.248257] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9704.139269] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9704.577617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9710.777889] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9711.064557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9715.683901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.690327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9715.690351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9715.693444] Lustre: Skipped 2 previous similar messages [ 9715.735868] Lustre: Skipped 1 previous similar message [ 9719.449672] Lustre: server umount lustre-OST0000 complete [ 9725.408723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9725.416309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.435422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.151339] Lustre: server umount lustre-MDT0000 complete [ 9731.042024] LustreError: 305496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9731.055831] LustreError: 305496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9733.821578] LustreError: 305482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025870 with bad export cookie 12471861327914154632 [ 9733.824621] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9733.832453] LustreError: 305482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9734.127900] Lustre: server umount lustre-MDT0001 complete [ 9743.994145] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:37:59 (1789025879) [ 9745.558988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9747.423221] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:38:02 (1789025882) [ 9750.954621] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9782.131715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.439783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9782.463983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9782.657157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9787.881432] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9796.416871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9796.554645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9796.619666] Lustre: 311317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9802.156292] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9824.049544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9824.274512] Lustre: Denying initial registration attempt from nid 192.168.202.138@tcp, specified as failover [ 9824.286760] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9824.295909] LustreError: 312771:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9824.307866] LustreError: 312771:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9824.318818] LustreError: 312771:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9824.421786] Lustre: server umount lustre-OST0000 complete [ 9824.425917] LustreError: 312771:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9832.933448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9832.943134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.957528] Lustre: Skipped 1 previous similar message [ 9832.987483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9833.001911] Lustre: Skipped 1 previous similar message [ 9837.978283] Lustre: server umount lustre-MDT0000 complete [ 9840.102547] LustreError: 312443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.113990] LustreError: 312443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9841.714458] LustreError: 311311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025978 with bad export cookie 12471861327914155514 [ 9841.741615] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9842.201569] Lustre: server umount lustre-MDT0001 complete [ 9858.157491] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 9862.177328] Key type lgssc unregistered [ 9862.750562] LNet: 313894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9862.763786] LNetError: 313894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9862.782702] LNet: Removed LNI 192.168.202.138@tcp [ 9864.408341] Key type .llcrypt unregistered [ 9864.415094] Key type ._llcrypt unregistered [ 9881.575250] Key type ._llcrypt registered [ 9881.577196] Key type .llcrypt registered [ 9881.670799] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9882.773433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9882.788575] alg: No test for adler32 (adler32-zlib) [ 9884.139752] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 9884.583523] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 9886.295928] Key type lgssc registered [ 9887.415858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9941.560264] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 9952.860066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9952.889993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.138257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9954.157782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9954.262111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.507414] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9967.395509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.461884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9967.533537] Lustre: 316659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9967.806350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.964722] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 9979.109763] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9984.883278] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10004.549966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.788399] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10004.993196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10012.724484] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10020.546411] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10025.445359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10025.452221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.479053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10025.952393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10025.960073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.079481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10031.085449] Lustre: Skipped 2 previous similar messages [10036.194462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10036.200065] Lustre: Skipped 1 previous similar message [10037.215066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10037.405690] Lustre: server umount lustre-OST0000 complete [10044.896910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10044.909337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.942318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.397282] Lustre: server umount lustre-MDT0000 complete [10052.328919] LustreError: 316650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026188 with bad export cookie 13304404885870508123 [10052.338483] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10052.346206] LustreError: 316650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10052.789557] Lustre: server umount lustre-MDT0001 complete [10063.444396] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10066.433266] Key type lgssc unregistered [10066.760878] LNet: 320233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10066.767085] LNetError: 320233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10066.780753] LNet: Removed LNI 192.168.202.138@tcp [10067.547913] Key type .llcrypt unregistered [10067.550521] Key type ._llcrypt unregistered [10092.626778] Key type ._llcrypt registered [10092.630488] Key type .llcrypt registered [10092.734714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10093.898316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10094.052272] alg: No test for adler32 (adler32-zlib) [10095.149172] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10095.369376] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10097.047354] Key type lgssc registered [10098.355465] Lustre: Echo OBD driver; http://www.lustre.org/ [10106.081936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10106.098860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.581272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.061767] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10118.114309] LustreError: 321646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.134331] LustreError: 321646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10120.656624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.263834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10126.619952] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10134.005496] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10139.542829] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10148.330601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10148.821339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10155.395299] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10163.184836] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10177.505552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10177.514859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.534534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10181.221597] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [10181.607370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.329586] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [10186.344343] Lustre: Skipped 3 previous similar messages [10191.328466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10191.430087] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [10191.440749] Lustre: Skipped 3 previous similar messages [10191.844094] LustreError: 321646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.860434] LustreError: 321646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10191.927379] Lustre: server umount lustre-MDT0000 complete [10195.331723] LustreError: 321628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026331 with bad export cookie 7871092165972789302 [10195.340931] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.344761] LustreError: 321628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10196.549170] LustreError: 321641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.573531] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [10196.964370] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.983109] Lustre: Skipped 1 previous similar message [10200.999280] LustreError: 321643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10201.024818] LustreError: 321643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10201.792919] Lustre: server umount lustre-MDT0001 complete [10209.126191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10217.982822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10220.127146] LustreError: 325245:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10220.138330] LustreError: 325245:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000007f94b661 x1875929969232512/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789026356 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10220.163260] LustreError: 325245:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10220.524485] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6d3bbc9f432b244b [10220.544577] Lustre: MGC192.168.202.138@tcp: Connection restored to 0@lo (at 0@lo) [10220.877627] LustreError: 323479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.909547] LustreError: 323479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10220.965107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10221.043643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10225.203330] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10230.202990] LustreError: 323480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.212888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10230.215107] LustreError: 323480:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10233.564861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.970535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10239.492367] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10239.636329] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10248.769249] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10250.710989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10257.877230] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10259.457590] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10265.057561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.074055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10265.077596] Lustre: Skipped 1 previous similar message [10265.109939] Lustre: Skipped 3 previous similar messages [10279.392056] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10279.730180] Lustre: server umount lustre-OST0000 complete [10285.028663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10285.042640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.073082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.081675] Lustre: Skipped 4 previous similar messages [10298.335441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10298.617630] Lustre: server umount lustre-MDT0000 complete [10300.896944] LustreError: 325264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.928986] LustreError: 325264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10303.034498] LustreError: 321629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026439 with bad export cookie 7871092165972796491 [10303.051740] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10303.725355] Lustre: server umount lustre-MDT0001 complete [10317.725548] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10320.758310] Key type lgssc unregistered [10321.205533] LNet: 327935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10321.228914] LNetError: 327935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10322.278650] LNet: Removed LNI 192.168.202.138@tcp [10322.870333] Key type .llcrypt unregistered [10322.874279] Key type ._llcrypt unregistered [10346.490970] Key type ._llcrypt registered [10346.494915] Key type .llcrypt registered [10346.607829] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10347.877267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10348.074388] alg: No test for adler32 (adler32-zlib) [10349.276864] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10349.705491] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10351.431681] Key type lgssc registered [10352.980427] Lustre: Echo OBD driver; http://www.lustre.org/ [10362.432565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10362.477202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.385118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10370.087289] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10373.603894] LustreError: 329349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.626354] LustreError: 329349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10378.721795] LustreError: 329348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.085642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.752869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10388.599079] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10396.344055] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10403.056974] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10408.421599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.422516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10408.443340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.478908] Lustre: Skipped 1 previous similar message [10409.505375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10409.524207] Lustre: Skipped 1 previous similar message [10412.851857] Lustre: server umount lustre-MDT0000 complete [10414.562598] LustreError: 330079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.580026] LustreError: 330079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10416.531852] LustreError: 329329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026553 with bad export cookie 10535768094614564369 [10416.555422] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10417.006665] Lustre: server umount lustre-MDT0001 complete [10432.427259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.649629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10432.688799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10433.047806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10437.933983] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10447.273994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.381710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10447.433616] Lustre: 331938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10447.698116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10451.968549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10459.599344] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.019932] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10475.834471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10476.148645] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10476.168595] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10476.187664] LustreError: 333823:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10476.198781] LustreError: 333823:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10476.208202] LustreError: 333823:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10476.285303] Lustre: server umount lustre-OST0000 complete [10476.288794] LustreError: 333823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10485.948925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.219973] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10486.385395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10492.642729] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10495.114095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10499.295877] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10506.030824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.137076] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10506.266149] Lustre: lustre-OST0001: new disk, initializing [10506.275195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10506.279772] Lustre: 335363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10506.390505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10514.047445] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10516.577155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10516.588687] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10516.728596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10521.433730] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10527.212475] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10527.214139] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.215869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10527.255537] Lustre: Skipped 2 previous similar messages [10530.496392] Lustre: server umount lustre-OST0001 complete [10532.323118] LustreError: 334778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.352989] LustreError: 334778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10535.905236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10535.921567] LustreError: Skipped 1 previous similar message [10535.931977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.955483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10535.963793] Lustre: Skipped 1 previous similar message [10537.442376] LustreError: 334292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.512171] LustreError: 334292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10542.569679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10542.586789] Lustre: Skipped 4 previous similar messages [10547.696516] LustreError: 336167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.715356] LustreError: 336167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10549.216737] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10549.415757] Lustre: server umount lustre-OST0000 complete [10555.360244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10555.365851] LustreError: Skipped 1 previous similar message [10555.369169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.376100] Lustre: Skipped 1 previous similar message [10555.380142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10555.384227] Lustre: Skipped 2 previous similar messages [10559.823164] Lustre: server umount lustre-MDT0000 complete [10563.623780] LustreError: 331931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026700 with bad export cookie 10535768094614564922 [10563.633640] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10563.635562] LustreError: 331931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10564.064067] Lustre: server umount lustre-MDT0001 complete [10576.712115] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [10579.412980] Key type lgssc unregistered [10579.672337] LNet: 337321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10579.681496] LNetError: 337321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10580.709756] LNet: Removed LNI 192.168.202.138@tcp [10581.811635] Key type .llcrypt unregistered [10581.817900] Key type ._llcrypt unregistered [10671.822139] Key type ._llcrypt registered [10671.824678] Key type .llcrypt registered [10671.987913] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [10690.362216] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [10691.722577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10691.744923] alg: No test for adler32 (adler32-zlib) [10692.952386] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10693.311204] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [10695.001841] Key type lgssc registered [10696.510785] Lustre: Echo OBD driver; http://www.lustre.org/ [10734.531333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10734.580960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.959068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10735.989859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10736.096804] Lustre: lustre-MDT0000: new disk, initializing [10736.162379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.190359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10740.970896] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10752.103192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.291165] Lustre: 342257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10752.394645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10752.403792] Lustre: Skipped 1 previous similar message [10752.473296] Lustre: lustre-MDT0001: new disk, initializing [10752.559236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10752.614446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10752.624072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10758.627779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10768.533644] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10775.183193] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10783.278590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10783.669077] Lustre: lustre-OST0000: new disk, initializing [10783.686251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10783.695736] Lustre: 344227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10783.831428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10790.595970] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10790.986233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10790.996180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10791.107443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10800.142243] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10805.425504] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10805.704127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10810.375805] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10810.666777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10817.010424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10817.016531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.026780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10817.030387] Lustre: Skipped 1 previous similar message [10817.046742] Lustre: Skipped 1 previous similar message [10822.129156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10822.134154] Lustre: Skipped 1 previous similar message [10827.232862] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10827.242978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10827.261148] Lustre: Skipped 1 previous similar message [10827.548257] Lustre: server umount lustre-OST0000 complete [10832.364613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10832.369914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10832.380748] Lustre: Skipped 1 previous similar message [10832.402389] Lustre: Skipped 1 previous similar message [10837.594458] Lustre: server umount lustre-MDT0000 complete [10841.466483] LustreError: 342249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026977 with bad export cookie 1636806622315390879 [10841.467661] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10841.481836] LustreError: 342249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10841.887299] Lustre: server umount lustre-MDT0001 complete [10850.186778] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:56:25 (1789026985) [10857.020609] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:56:32 (1789026992) [10866.097882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.631674] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.651870] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10866.736668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10871.192883] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10871.782093] LustreError: 346984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.900688] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.494382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.951184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10885.454346] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10893.492819] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10899.510376] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10906.726139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10907.215738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10914.023605] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10925.001053] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10933.177303] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10935.211693] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10937.644582] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10940.379200] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10943.288085] Lustre: Failing over lustre-MDT0000 [10944.050740] Lustre: server umount lustre-MDT0000 complete [10946.528903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.553485] Lustre: Skipped 1 previous similar message [10946.564809] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.582395] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10951.649837] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.688749] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10961.892466] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.914633] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10962.913801] Lustre: 340482:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789027083/real 1789027083] req@0000000046a3e35e x1875930596320768/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789027099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10962.965327] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10965.716991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.153353] LustreError: 340478:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000046a3e35e x1875930596326400/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10973.624252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10973.696198] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10977.868758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10978.815710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10978.870352] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10978.967856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10979.618411] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [10988.847622] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10990.285512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10994.094615] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10996.629868] Lustre: Failing over lustre-MDT0000 [10997.149347] Lustre: server umount lustre-MDT0000 complete [10998.377634] LustreError: 346980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.407031] LustreError: 346980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10999.267746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.273128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10999.282641] Lustre: Skipped 1 previous similar message [10999.296199] LustreError: Skipped 1 previous similar message [11015.119088] Lustre: 340480:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789027135/real 1789027135] req@0000000017ec99ee x1875930596342272/t0(0) o400->MGC192.168.202.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789027151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11015.187647] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11016.605057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.611894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11025.658558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11030.087336] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11030.744946] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11031.034703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11031.046323] Lustre: Skipped 2 previous similar messages [11031.102475] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11031.183607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11041.589137] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11043.954175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11051.491616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11051.504083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.511777] Lustre: Skipped 3 previous similar messages [11051.512545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11051.536142] Lustre: Skipped 2 previous similar messages [11057.904606] Lustre: server umount lustre-MDT0000 complete [11061.729516] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.768974] LustreError: 347711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 29 previous similar messages [11061.975373] LustreError: 346965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027198 with bad export cookie 1636806622315393595 [11061.988141] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11061.992559] LustreError: 346965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11062.508861] Lustre: server umount lustre-MDT0001 complete [11077.511601] Lustre: server umount lustre-OST0000 complete [11096.690517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.603300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11104.176210] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11114.109060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.347738] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11128.389668] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11135.206888] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11142.570083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11149.594616] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11157.120930] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11167.474189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11172.833621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.838212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11172.845555] Lustre: Skipped 1 previous similar message [11172.858301] Lustre: Skipped 4 previous similar messages [11174.473432] Lustre: server umount lustre-OST0000 complete [11181.025883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11185.031294] Lustre: server umount lustre-MDT0000 complete [11188.194645] LustreError: 353332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.216764] LustreError: 353332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11188.475204] LustreError: 355191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027324 with bad export cookie 1636806622315394337 [11188.480060] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11188.486515] LustreError: 355191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11188.915797] Lustre: server umount lustre-MDT0001 complete [11200.506752] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11203.946439] Key type lgssc unregistered [11204.348581] LNet: 356897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11204.357908] LNetError: 356897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11204.392388] LNet: Removed LNI 192.168.202.138@tcp [11205.663174] Key type .llcrypt unregistered [11205.668784] Key type ._llcrypt unregistered [11235.193582] Key type ._llcrypt registered [11235.196621] Key type .llcrypt registered [11235.333112] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11236.691054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11236.739965] alg: No test for adler32 (adler32-zlib) [11238.054692] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11238.380745] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11240.079212] Key type lgssc registered [11241.368161] Lustre: Echo OBD driver; http://www.lustre.org/ [11249.624426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11249.648135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11251.047857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11256.023392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11261.420104] LustreError: 358412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.444180] LustreError: 358412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11265.230927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.732788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11270.421845] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11278.550771] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11284.286949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11291.915652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11292.303914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11296.456056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11298.960755] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11306.590699] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11322.337288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11322.349432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.371491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11323.896217] Lustre: server umount lustre-MDT0000 complete [11326.443267] LustreError: 359140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.446176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.491897] Lustre: Skipped 1 previous similar message [11327.720156] LustreError: 358393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027464 with bad export cookie 12770819056888281133 [11327.749320] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.128449] Lustre: server umount lustre-MDT0001 complete [11342.890529] Lustre: server umount lustre-OST0000 complete [11356.498567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.951289] LustreError: 362144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.976954] LustreError: 362144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11357.049945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11361.448944] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11362.274990] LustreError: 362145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11362.308405] LustreError: 362145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11367.395904] LustreError: 362144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11370.497303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.922471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11375.646294] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11384.872862] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11391.524574] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11399.715857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11400.005652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.044355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11408.240541] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11417.862717] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11432.417210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11432.432433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.468290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.444280] Lustre: server umount lustre-MDT0000 complete [11438.050395] LustreError: 362872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.075789] LustreError: 362872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11440.913225] LustreError: 362125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027577 with bad export cookie 12770819056888282246 [11440.914080] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11440.933411] LustreError: 362125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11441.478900] Lustre: server umount lustre-MDT0001 complete [11455.062614] Lustre: server umount lustre-OST0000 complete [11472.144929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.776956] LustreError: 366056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.862367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.987101] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11490.178369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11497.616980] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11506.296405] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11512.630257] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11519.900644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11520.302479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11520.310957] Lustre: Skipped 1 previous similar message [11527.264768] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11535.438578] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11542.839098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11556.320874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11556.328853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11556.340240] Lustre: Skipped 2 previous similar messages [11556.359313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11556.361653] Lustre: Skipped 2 previous similar messages [11556.577865] Lustre: server umount lustre-OST0000 complete [11562.471609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11562.478927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.504327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11568.612811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11568.624729] Lustre: Skipped 3 previous similar messages [11575.264559] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11575.559831] Lustre: server umount lustre-MDT0000 complete [11577.824663] LustreError: 366052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.851088] LustreError: 366052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11580.146453] LustreError: 366037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027716 with bad export cookie 12770819056888283268 [11580.154938] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.168240] LustreError: 366037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11580.673960] Lustre: server umount lustre-MDT0001 complete [11593.751425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11596.672072] Key type lgssc unregistered [11597.025703] LNet: 369610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11597.032820] LNetError: 369610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11597.043859] LNet: Removed LNI 192.168.202.138@tcp [11597.871480] Key type .llcrypt unregistered [11597.874060] Key type ._llcrypt unregistered [11629.014494] Key type ._llcrypt registered [11629.029396] Key type .llcrypt registered [11629.220370] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11630.255339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11630.347246] alg: No test for adler32 (adler32-zlib) [11631.461081] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11631.828127] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11633.567249] Key type lgssc registered [11635.409564] Lustre: Echo OBD driver; http://www.lustre.org/ [11645.622057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11645.655361] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.662182] LustreError: 371050:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11645.676340] LustreError: 371050:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11645.688463] LustreError: 371050:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11645.706061] LustreError: 371050:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11645.717226] LustreError: 371050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11656.192879] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11656.206126] LustreError: 371447:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11656.238683] LustreError: 371447:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11656.246777] LustreError: 371447:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11656.276723] LustreError: 371447:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11656.295114] LustreError: 371447:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11684.387627] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [11687.336888] Key type lgssc unregistered [11687.866371] LNet: 372470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11687.871037] LNetError: 372470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11687.902313] LNet: Removed LNI 192.168.202.138@tcp [11689.043819] Key type .llcrypt unregistered [11689.050830] Key type ._llcrypt unregistered [11712.713047] Key type ._llcrypt registered [11712.719334] Key type .llcrypt registered [11712.854759] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [11730.278807] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [11731.750612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11731.799969] alg: No test for adler32 (adler32-zlib) [11732.778279] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11733.228754] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [11734.983304] Key type lgssc registered [11736.664800] Lustre: Echo OBD driver; http://www.lustre.org/ [11775.000216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11775.058233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.396587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11776.440848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11776.550699] Lustre: lustre-MDT0000: new disk, initializing [11776.686914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11776.712915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11781.750554] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11792.661145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.743469] Lustre: 375623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11792.783922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11792.787171] Lustre: Skipped 1 previous similar message [11792.841544] Lustre: lustre-MDT0001: new disk, initializing [11792.924751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11792.957424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11792.962718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11797.418548] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11807.263197] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11812.725515] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11820.050168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11820.431644] Lustre: lustre-OST0000: new disk, initializing [11820.437724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11820.448399] Lustre: 377592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11820.575178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11825.792515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11825.808067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11825.946051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11828.053443] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11839.012381] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11843.568213] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11843.875875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11849.169318] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11849.406434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11851.745627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11851.748115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.757234] LustreError: Skipped 1 previous similar message [11851.759401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11851.778534] Lustre: Skipped 1 previous similar message [11851.813437] Lustre: Skipped 1 previous similar message [11856.870771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11856.878305] Lustre: Skipped 1 previous similar message [11857.791548] Lustre: server umount lustre-OST0000 complete [11861.985713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.991199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.999288] Lustre: Skipped 1 previous similar message [11867.106816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11867.119940] Lustre: Skipped 2 previous similar messages [11867.737358] Lustre: server umount lustre-MDT0000 complete [11871.530783] LustreError: 375618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028008 with bad export cookie 16300607066730196033 [11871.537787] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11871.547690] LustreError: 375618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11872.102560] Lustre: server umount lustre-MDT0001 complete [11880.988363] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11882.324411] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:13:37 (1789028017) [11888.766464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.322830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11893.922541] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11894.753476] LustreError: 380050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.785073] LustreError: 380050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11898.859359] LustreError: 380051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.932275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11903.279915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11908.461769] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11916.057481] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11922.714986] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11930.854906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11931.414258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11939.553459] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11949.936285] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11959.545200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11959.821306] Lustre: lustre-OST0001: new disk, initializing [11959.824663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11959.830255] Lustre: 382888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11959.921038] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11966.655765] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [11968.126801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11968.138088] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11968.244065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11976.949708] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11983.851672] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.865130] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11983.886373] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11983.914433] Lustre: Skipped 1 previous similar message [11987.438134] Lustre: server umount lustre-OST0001 complete [11988.969365] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.990899] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11994.085454] LustreError: 383321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.103855] LustreError: 383321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11999.230363] LustreError: 381874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.253991] LustreError: 381874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12006.368646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12006.378563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12006.401206] Lustre: Skipped 1 previous similar message [12006.412358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12006.423555] Lustre: Skipped 1 previous similar message [12009.446483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12009.467878] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.496966] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12020.191187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12020.601113] Lustre: server umount lustre-OST0000 complete [12026.349735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12026.371236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.395539] Lustre: Skipped 1 previous similar message [12026.405326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12026.415160] Lustre: Skipped 6 previous similar messages [12031.254535] Lustre: server umount lustre-MDT0000 complete [12035.045587] LustreError: 381292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12035.059588] LustreError: 381292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12035.172712] LustreError: 380031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028171 with bad export cookie 16300607066730196915 [12035.177272] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12035.178696] LustreError: 380031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12041.554715] Lustre: server umount lustre-MDT0001 complete [12052.497449] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12054.946302] Key type lgssc unregistered [12055.332213] LNet: 384876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12055.336282] LNetError: 384876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12056.359780] LNet: Removed LNI 192.168.202.138@tcp [12057.251545] Key type .llcrypt unregistered [12057.253290] Key type ._llcrypt unregistered [12113.516679] Key type ._llcrypt registered [12113.533613] Key type .llcrypt registered [12113.714813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12114.795414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12114.807236] alg: No test for adler32 (adler32-zlib) [12115.940108] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12116.152806] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12117.783336] Key type lgssc registered [12118.987207] Lustre: Echo OBD driver; http://www.lustre.org/ [12126.881278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12126.914764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.260842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12128.294463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12128.524243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12133.412757] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12142.580412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12142.741811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12142.802773] Lustre: 387671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12143.044656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.561409] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12157.526603] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12164.629930] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12168.676332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12168.685540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12168.694417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.703488] Lustre: Skipped 1 previous similar message [12173.042659] Lustre: server umount lustre-MDT0000 complete [12173.792938] LustreError: 387679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12173.801492] LustreError: 387679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12175.661625] LustreError: 387663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028312 with bad export cookie 5789636366257304532 [12175.674604] LustreError: 387663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12175.676606] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12176.109844] Lustre: server umount lustre-MDT0001 complete [12185.877544] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:18:41 (1789028321) [12192.186724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12192.541235] LustreError: 390185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.600599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12196.944583] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12197.859084] LustreError: 390186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12202.982028] LustreError: 390185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.662869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.077437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.472363] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12215.093949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12220.546934] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12226.350379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12226.506811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12226.663813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12232.010628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12238.424781] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12246.093916] Lustre: Permanently deactivating lustre-OST0000 [12246.124559] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12253.747906] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12256.148839] LustreError: 393171:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12256.155473] LustreError: 393171:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12256.159979] LustreError: 393171:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12259.713914] LustreError: 393221:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12259.719499] LustreError: 393221:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12264.241777] Lustre: server umount lustre-OST0000 complete [12267.747958] LustreError: 393470:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12267.756557] LustreError: 393470:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12271.076598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12271.085774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12271.095219] Lustre: Skipped 2 previous similar messages [12271.113325] Lustre: Skipped 1 previous similar message [12276.197671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12276.206865] Lustre: Skipped 1 previous similar message [12281.312157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12281.326399] Lustre: Skipped 1 previous similar message [12284.385758] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12284.628834] Lustre: server umount lustre-MDT0000 complete [12286.433837] LustreError: 392329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12286.471965] LustreError: 392329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12289.000654] LustreError: 390165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028425 with bad export cookie 5789636366257305113 [12289.008414] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12289.009881] LustreError: 390165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12299.312292] Lustre: server umount lustre-MDT0001 complete [12310.263174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.413948] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12317.260221] LustreError: 394632:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12323.723558] Lustre: 394970:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.138@tcp [12326.060183] Lustre: server umount MGS complete [12337.720357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.229468] LustreError: 395696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.247913] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12338.255981] Lustre: Skipped 1 previous similar message [12338.334257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12344.319712] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12354.890926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12355.095141] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12355.153330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12359.164781] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12367.622570] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12374.693181] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12377.181391] Lustre: Permanently reactivating lustre-OST0000 [12377.213337] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12377.237937] Lustre: Skipped 2 previous similar messages [12403.679232] LustreError: 397428:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12411.348564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12411.691761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12418.364512] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12428.232806] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12433.380481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12433.384054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12433.460147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12434.403544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12438.167108] Lustre: server umount lustre-OST0000 complete [12442.081540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12442.087843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.095140] Lustre: Skipped 1 previous similar message [12442.098683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12442.101764] Lustre: Skipped 2 previous similar messages [12448.135063] Lustre: server umount lustre-MDT0000 complete [12449.760353] LustreError: 398686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12449.780441] LustreError: 398686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12451.780532] LustreError: 395676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028588 with bad export cookie 5789636366257306163 [12451.794144] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12451.797052] LustreError: 395676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12452.147731] Lustre: server umount lustre-MDT0001 complete [12462.557574] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12465.346767] Key type lgssc unregistered [12465.799857] LNet: 400009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12465.813897] LNetError: 400009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12465.840383] LNet: Removed LNI 192.168.202.138@tcp [12466.803669] Key type .llcrypt unregistered [12466.806349] Key type ._llcrypt unregistered [12485.214340] Key type ._llcrypt registered [12485.217447] Key type .llcrypt registered [12485.336911] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [12503.885462] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12505.042955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12505.072221] alg: No test for adler32 (adler32-zlib) [12506.149748] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12506.397141] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12508.071640] Key type lgssc registered [12509.220828] Lustre: Echo OBD driver; http://www.lustre.org/ [12544.214174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12544.260568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.567562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12545.629537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12545.761270] Lustre: lustre-MDT0000: new disk, initializing [12545.894480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12545.921916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12551.422773] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12563.521425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12563.670568] Lustre: 403172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12563.757902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12563.766157] Lustre: Skipped 1 previous similar message [12563.893211] Lustre: lustre-MDT0001: new disk, initializing [12564.037105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12564.082620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12564.098292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12569.231594] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12578.952424] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12585.238733] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12592.380945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12592.821482] Lustre: lustre-OST0000: new disk, initializing [12592.826126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12592.836563] Lustre: 405140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12592.959413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12600.658549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12601.894822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12601.920767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12602.052818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12612.085430] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12617.093664] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12617.435490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12621.821872] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12621.996958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12627.939073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12627.949099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12627.962645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12630.350284] Lustre: server umount lustre-OST0000 complete [12635.619218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12635.635765] LustreError: Skipped 1 previous similar message [12635.650029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.668685] Lustre: Skipped 1 previous similar message [12635.680452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12635.682508] Lustre: Skipped 1 previous similar message [12638.176870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.181480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12638.204362] Lustre: Skipped 1 previous similar message [12640.562884] Lustre: server umount lustre-MDT0000 complete [12644.456568] LustreError: 403164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028780 with bad export cookie 5737696775222775596 [12644.457199] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12644.462924] LustreError: 403164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12644.709882] Lustre: server umount lustre-MDT0001 complete [12653.520799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12655.153480] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:26:30 (1789028790) [12662.912219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12663.687889] LustreError: 407599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.714673] LustreError: 407599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12663.901986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12668.898321] LustreError: 407600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.586327] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12674.021263] LustreError: 407599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12679.136366] LustreError: 407600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.182799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.682792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12687.507562] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12695.498785] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12701.943868] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12710.188388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12710.754015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.295477] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12728.230655] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12730.509506] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12739.016217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12748.769462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12748.785135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12748.814602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12751.370275] Lustre: server umount lustre-OST0000 complete [12759.520877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12759.535213] LustreError: Skipped 1 previous similar message [12759.541903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.553337] Lustre: Skipped 1 previous similar message [12759.563418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12759.565988] Lustre: Skipped 2 previous similar messages [12763.200232] Lustre: server umount lustre-MDT0000 complete [12765.161205] LustreError: 407594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.197994] LustreError: 407594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12768.090285] LustreError: 407581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028904 with bad export cookie 5737696775222776471 [12768.112932] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12768.114942] LustreError: 407581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12769.087706] Lustre: server umount lustre-MDT0001 complete [12783.781554] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12787.114774] Key type lgssc unregistered [12787.600624] LNet: 411327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12787.605499] LNetError: 411327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12787.646087] LNet: Removed LNI 192.168.202.138@tcp [12788.759175] Key type .llcrypt unregistered [12788.761313] Key type ._llcrypt unregistered [12817.244164] Key type ._llcrypt registered [12817.249413] Key type .llcrypt registered [12817.404366] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12817.732375] Key type .llcrypt unregistered [12817.736468] Key type ._llcrypt unregistered [12840.789165] Key type ._llcrypt registered [12840.790469] Key type .llcrypt registered [12840.940831] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12842.433223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12842.686716] alg: No test for adler32 (adler32-zlib) [12843.686113] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12844.037178] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12845.839214] Key type lgssc registered [12847.658457] Lustre: Echo OBD driver; http://www.lustre.org/ [12858.319356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12858.357803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12860.557853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12865.098839] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12871.143496] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12871.167101] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12873.919262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12874.255385] LustreError: 413539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12874.316892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12879.331908] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.363027] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12881.028603] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12884.449413] LustreError: 413539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12884.463242] LustreError: 413539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12886.563269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12888.408659] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12891.616541] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.641940] LustreError: 413538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12895.152634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.456438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12900.299881] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [12910.125277] Lustre: server umount lustre-OST0000 complete [12916.204938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12916.215504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.240158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.260088] Lustre: Skipped 1 previous similar message [12921.313430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12924.509151] Lustre: server umount lustre-MDT0000 complete [12926.434721] LustreError: 413539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.457291] LustreError: 413539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12927.519298] Lustre: 412848:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789029047/real 1789029047] req@00000000aa276dec x1875932851759232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789029063 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12927.550862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12927.559910] Lustre: Skipped 1 previous similar message [12928.428223] LustreError: 415644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029064 with bad export cookie 13524415477529297350 [12928.439811] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12928.442521] LustreError: 415644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12928.702118] Lustre: server umount lustre-MDT0001 complete [12940.757596] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [12943.687090] Key type lgssc unregistered [12943.963322] LNet: 416684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12943.970382] LNetError: 416684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12943.988888] LNet: Removed LNI 192.168.202.138@tcp [12944.869236] Key type .llcrypt unregistered [12944.874298] Key type ._llcrypt unregistered [12970.260361] Key type ._llcrypt registered [12970.264692] Key type .llcrypt registered [12970.408524] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [12971.928674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12971.981548] alg: No test for adler32 (adler32-zlib) [12973.082287] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12973.362316] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [12975.023273] Key type lgssc registered [12976.525271] Lustre: Echo OBD driver; http://www.lustre.org/ [12984.427128] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12984.453662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13000.031483] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13005.151735] LustreError: 418069:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [13031.775764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13037.941491] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13045.253772] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13051.963936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.476068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13052.554084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13057.272853] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13057.505142] LustreError: 418095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13057.519182] LustreError: 418095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13062.636860] LustreError: 418094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.547227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13066.728458] LustreError: 418095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.745187] LustreError: 418095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13066.942241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13071.527788] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13079.114230] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13084.324894] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13092.834428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13092.838624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13092.857142] Lustre: Skipped 1 previous similar message [13092.873871] Lustre: Skipped 1 previous similar message [13093.967464] Lustre: server umount lustre-OST0000 complete [13098.302741] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13098.315302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13098.328398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13098.509967] Lustre: server umount lustre-MDT0000 complete [13102.401399] LustreError: 418076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029238 with bad export cookie 16774306465688605067 [13102.418428] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13102.420479] LustreError: 418076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13102.846280] Lustre: server umount lustre-MDT0001 complete [13113.526425] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13116.478847] Key type lgssc unregistered [13116.737843] LNet: 421636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13116.748438] LNetError: 421636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13117.807068] LNet: Removed LNI 192.168.202.138@tcp [13118.713927] Key type .llcrypt unregistered [13118.716335] Key type ._llcrypt unregistered [13143.086990] Key type ._llcrypt registered [13143.089270] Key type .llcrypt registered [13143.203403] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13144.082684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13144.168110] alg: No test for adler32 (adler32-zlib) [13145.385156] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [13145.623847] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13147.335257] Key type lgssc registered [13148.842502] Lustre: Echo OBD driver; http://www.lustre.org/ [13157.464950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13157.502182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13159.073958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13165.121492] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13168.621521] LustreError: 423052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.654443] LustreError: 423052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13173.731902] LustreError: 423051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13174.431063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13175.133883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13179.547375] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13187.525280] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13193.213662] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13201.209664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13201.640825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13203.709097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13208.480126] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13218.336152] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13226.091765] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.38@tcp (stopping) [13226.466180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13226.496531] Lustre: Skipped 2 previous similar messages [13230.157866] Lustre: server umount lustre-MDT0000 complete [13231.185303] LustreError: 425427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.112719] LustreError: 423033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.38@tcp arrived at 1789029369 with bad export cookie 8391175905081177198 [13233.131316] LustreError: 423033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13235.177223] LustreError: 425633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029371 with bad export cookie 8391175905081176309 [13235.186049] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13235.192203] LustreError: 425633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13236.707211] LustreError: 423052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.713778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13236.730157] LustreError: 423052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13236.764304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13236.768753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13236.781782] Lustre: Skipped 3 previous similar messages [13240.739351] LustreError: 425427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13240.743226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13240.756785] LustreError: 425427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13241.473500] Lustre: server umount lustre-OST0000 complete [13248.107176] Lustre: server umount lustre-MDT0001 complete [13259.647338] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13263.187549] Key type lgssc unregistered [13263.544171] LNet: 426739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13263.568805] LNetError: 426739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13263.605138] LNet: Removed LNI 192.168.202.138@tcp [13264.641184] Key type .llcrypt unregistered [13264.644128] Key type ._llcrypt unregistered [13291.926156] Key type ._llcrypt registered [13291.929298] Key type .llcrypt registered [13292.071837] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13293.601131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13294.072141] alg: No test for adler32 (adler32-zlib) [13295.211211] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [13295.652596] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13297.431987] Key type lgssc registered [13298.526135] Lustre: Echo OBD driver; http://www.lustre.org/ [13309.471158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13309.491369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.181646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13317.389754] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13320.682167] LustreError: 428154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13320.708626] LustreError: 428154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13325.796436] LustreError: 428155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.597361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13330.418816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13336.632458] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13344.898687] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13352.563692] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13360.293422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13360.795742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13368.265403] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13378.312079] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13381.291396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13389.412217] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [13391.840747] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13391.850041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.861191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13391.884467] Lustre: Skipped 1 previous similar message [13391.893730] Lustre: Skipped 1 previous similar message [13393.658789] Lustre: server umount lustre-MDT0001 complete [13394.519608] LustreError: 428150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13396.968207] LustreError: 428154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13402.093230] LustreError: 428884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13402.109018] LustreError: 428884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13407.209889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13407.248211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13409.143289] Lustre: server umount lustre-OST0000 complete [13412.333723] LustreError: 428884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13412.360686] LustreError: 428884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13414.043892] Lustre: server umount lustre-MDT0000 complete [13429.992586] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13433.859741] Key type lgssc unregistered [13434.187540] LNet: 431840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13434.198369] LNetError: 431840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13434.233914] LNet: Removed LNI 192.168.202.138@tcp [13435.985687] Key type .llcrypt unregistered [13435.988330] Key type ._llcrypt unregistered [13478.001795] Key type ._llcrypt registered [13478.005759] Key type .llcrypt registered [13478.187736] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [13500.330368] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13501.576569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13501.632430] alg: No test for adler32 (adler32-zlib) [13502.908651] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [13503.320456] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13505.111527] Key type lgssc registered [13506.511481] Lustre: Echo OBD driver; http://www.lustre.org/ [13552.280964] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13570.595300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13570.664639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13572.272257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13572.330570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13572.524360] Lustre: lustre-MDT0000: new disk, initializing [13572.598307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13572.626514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13579.501067] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13598.367682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13598.724653] Lustre: lustre-OST0000: new disk, initializing [13598.727729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13598.731236] Lustre: Skipped 1 previous similar message [13598.737730] Lustre: 436882:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13598.807450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13604.431808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13604.440391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13604.457119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13608.630805] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13621.212844] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13640.605371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13640.813733] Lustre: 435860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13640.899216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13641.105447] Lustre: lustre-MDT0001: new disk, initializing [13641.355307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13641.489779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13641.543654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13648.407974] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13659.033322] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13659.292259] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13664.495122] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13664.755263] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13670.894437] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13670.898127] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13670.900493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13670.923134] Lustre: Skipped 1 previous similar message [13674.605540] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.38@tcp (stopping) [13674.618056] Lustre: Skipped 1 previous similar message [13676.011249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13676.026765] Lustre: Skipped 1 previous similar message [13676.328233] Lustre: server umount lustre-MDT0001 complete [13686.240640] LustreError: 435871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13686.263776] LustreError: 435871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13691.369510] LustreError: 435867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13691.398921] LustreError: 435867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13696.484837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13696.487294] LustreError: 435872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13696.524204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13701.607157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13701.611382] LustreError: 435871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13706.729051] LustreError: 435872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13709.279569] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13709.751809] Lustre: server umount lustre-OST0000 complete [13721.120278] Lustre: server umount lustre-MDT0000 complete [13738.876697] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13742.510969] Key type lgssc unregistered [13742.777843] LNet: 440633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13742.789526] LNetError: 440633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13742.824326] LNet: Removed LNI 192.168.202.138@tcp [13743.956201] Key type .llcrypt unregistered [13743.960385] Key type ._llcrypt unregistered [13793.607162] Key type ._llcrypt registered [13793.613139] Key type .llcrypt registered [13793.873253] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [13794.314436] Key type .llcrypt unregistered [13794.317260] Key type ._llcrypt unregistered [13817.081697] Key type ._llcrypt registered [13817.084685] Key type .llcrypt registered [13817.216925] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [13818.648642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13818.822415] alg: No test for adler32 (adler32-zlib) [13820.103890] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [13820.334525] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [13822.063232] Key type lgssc registered [13824.497390] Lustre: Echo OBD driver; http://www.lustre.org/ [13860.347276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13860.435050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13861.747438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13861.773200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13861.897502] Lustre: lustre-MDT0000: new disk, initializing [13862.076185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13862.125198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13866.798802] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13880.222274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13880.459878] Lustre: 444007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13880.547061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13880.549946] Lustre: Skipped 1 previous similar message [13880.646445] Lustre: lustre-MDT0001: new disk, initializing [13880.750901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13880.794695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13880.810523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13887.676227] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13897.830578] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13904.657476] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13912.443770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13912.777265] Lustre: lustre-OST0000: new disk, initializing [13912.790174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13912.805107] Lustre: 445978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13912.944566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13921.449375] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13922.978651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13922.994617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13923.198057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13932.294882] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13938.658328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13938.661051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13938.696196] Lustre: Skipped 2 previous similar messages [13938.707067] Lustre: Skipped 2 previous similar messages [13943.782771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13943.786815] Lustre: Skipped 2 previous similar messages [13943.967979] Lustre: server umount lustre-MDT0000 complete [13948.256593] LustreError: 444001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030084 with bad export cookie 11414267333438625509 [13948.257253] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13948.274888] LustreError: 444001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13948.713561] Lustre: server umount lustre-MDT0001 complete [13963.421173] Lustre: server umount lustre-OST0000 complete [13976.495396] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:48:31 (1789030111) [13987.251758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13988.163816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13992.951369] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [13993.441562] LustreError: 448005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13993.458411] LustreError: 448005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13997.549915] LustreError: 448006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14002.658461] LustreError: 448005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14003.552257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14004.361152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14010.426015] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14019.539821] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14026.061263] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14047.404097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14047.749477] Lustre: Found index 0 for lustre-OST0000, updating log [14048.062829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14054.996138] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14059.411586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14064.792730] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14072.394606] Lustre: server umount lustre-OST0000 complete [14090.209175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14090.212788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14090.229787] Lustre: Skipped 1 previous similar message [14091.231396] Lustre: 442540:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789030211/real 1789030211] req@00000000f39185e5 x1875933875218560/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789030227 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14091.291562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14095.715777] Lustre: 442539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789030216/real 1789030216] req@00000000f437961e x1875933875219712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789030232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14095.735596] Lustre: 442539:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14096.355957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14096.359363] Lustre: Skipped 3 previous similar messages [14100.588804] Lustre: server umount lustre-MDT0000 complete [14100.901508] Lustre: 442539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789030221/real 1789030221] req@000000000e95eb5f x1875933875220864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789030237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14101.473304] LustreError: 448005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14101.519890] LustreError: 448005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14105.638208] Lustre: 442538:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789030226/real 1789030226] req@0000000087009917 x1875933875222656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789030242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14106.087796] LustreError: 447985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030242 with bad export cookie 11414267333438626902 [14106.095667] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14106.104336] LustreError: 447985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14106.664547] Lustre: server umount lustre-MDT0001 complete [14109.151483] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [14117.903909] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:50:53 (1789030253) [14750.113492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14750.903428] LustreError: 454433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14750.920019] LustreError: 454433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14751.045448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14755.721354] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14756.322332] LustreError: 454434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14756.341962] LustreError: 454434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14761.443788] LustreError: 454433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14765.398300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14765.877088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14770.825655] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14779.120855] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14785.795311] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14801.375276] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [14808.574601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14808.942396] Lustre: Found index 0 for lustre-OST0000, updating log [14809.313248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14816.671396] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [14824.830493] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14828.007798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14828.032388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14841.311155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14841.574298] Lustre: server umount lustre-OST0000 complete [14847.408470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14847.414588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14847.417069] Lustre: Skipped 1 previous similar message [14847.436922] Lustre: Skipped 1 previous similar message [14848.490437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14848.499029] Lustre: Skipped 1 previous similar message [14850.531542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14850.537254] Lustre: Skipped 3 previous similar messages [14852.670335] LustreError: 454434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14852.684211] LustreError: 454434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14852.712315] Lustre: server umount lustre-MDT0000 complete [14856.813046] LustreError: 454433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14856.838600] LustreError: 454433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [14857.346597] LustreError: 454414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030993 with bad export cookie 11414267333438627980 [14857.367708] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14857.740644] Lustre: server umount lustre-MDT0001 complete [14860.767223] LNet: 1 local NIs in recovery (showing 1): 192.168.202.138@tcp [14893.718775] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [14906.364585] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [14947.715839] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:04:43 (1789031083) [14966.886548] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [14970.033760] Key type lgssc unregistered [14970.501531] LNet: 461048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14970.511361] LNetError: 461048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15070.886408] LNet: Removed LNI 192.168.202.138@tcp [15072.069134] Key type .llcrypt unregistered [15072.071842] Key type ._llcrypt unregistered [15091.998497] Key type ._llcrypt registered [15092.003549] Key type .llcrypt registered [15092.121348] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [15093.455283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15093.746547] alg: No test for adler32 (adler32-zlib) [15095.031674] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [15095.407049] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [15097.135859] Key type lgssc registered [15098.628634] Lustre: Echo OBD driver; http://www.lustre.org/ [15104.041444] LNet: Added LNI 192.168.202.138@tcp100 [8/256/0/180] [15105.123815] LNet: Added LNI 192.168.202.138@tcp101 [8/256/0/180] [15106.018453] LNet: Added LNI 192.168.202.138@tcp102 [8/256/0/180] [15106.927857] LNet: Added LNI 192.168.202.138@tcp103 [8/256/0/180] [15107.803956] LNet: Added LNI 192.168.202.138@tcp104 [8/256/0/180] [15108.620675] LNet: Added LNI 192.168.202.138@tcp105 [8/256/0/180] [15109.645563] LNet: Added LNI 192.168.202.138@tcp106 [8/256/0/180] [15110.616365] LNet: Added LNI 192.168.202.138@tcp107 [8/256/0/180] [15111.465560] LNet: Added LNI 192.168.202.138@tcp108 [8/256/0/180] [15112.212819] LNet: Added LNI 192.168.202.138@tcp109 [8/256/0/180] [15113.120899] LNet: Added LNI 192.168.202.138@tcp110 [8/256/0/180] [15113.983332] LNet: Added LNI 192.168.202.138@tcp111 [8/256/0/180] [15114.768635] LNet: Added LNI 192.168.202.138@tcp112 [8/256/0/180] [15115.731855] LNet: Added LNI 192.168.202.138@tcp113 [8/256/0/180] [15116.643056] LNet: Added LNI 192.168.202.138@tcp114 [8/256/0/180] [15117.646662] LNet: Added LNI 192.168.202.138@tcp115 [8/256/0/180] [15118.603414] LNet: Added LNI 192.168.202.138@tcp116 [8/256/0/180] [15119.596645] LNet: Added LNI 192.168.202.138@tcp117 [8/256/0/180] [15120.457512] LNet: Added LNI 192.168.202.138@tcp118 [8/256/0/180] [15121.335657] LNet: Added LNI 192.168.202.138@tcp119 [8/256/0/180] [15122.318513] LNet: Added LNI 192.168.202.138@tcp120 [8/256/0/180] [15123.235865] LNet: Added LNI 192.168.202.138@tcp121 [8/256/0/180] [15124.180459] LNet: Added LNI 192.168.202.138@tcp122 [8/256/0/180] [15125.282913] LNet: Added LNI 192.168.202.138@tcp123 [8/256/0/180] [15126.164588] LNet: Added LNI 192.168.202.138@tcp124 [8/256/0/180] [15127.068736] LNet: Added LNI 192.168.202.138@tcp125 [8/256/0/180] [15128.071748] LNet: Added LNI 192.168.202.138@tcp126 [8/256/0/180] [15129.233116] LNet: Added LNI 192.168.202.138@tcp127 [8/256/0/180] [15130.058989] LNet: Added LNI 192.168.202.138@tcp128 [8/256/0/180] [15131.012911] LNet: Added LNI 192.168.202.138@tcp129 [8/256/0/180] [15132.011696] LNet: Added LNI 192.168.202.138@tcp130 [8/256/0/180] [15132.897697] LNet: Added LNI 192.168.202.138@tcp131 [8/256/0/180] [15133.850896] LNet: Added LNI 192.168.202.138@tcp132 [8/256/0/180] [15134.983963] LNet: Added LNI 192.168.202.138@tcp133 [8/256/0/180] [15135.923772] LNet: Added LNI 192.168.202.138@tcp134 [8/256/0/180] [15137.008709] LNet: Added LNI 192.168.202.138@tcp135 [8/256/0/180] [15144.521385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15144.550687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15145.711115] Lustre: 464074:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [15145.741564] LNetError: 464099:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15145.755987] LNetError: Protocol error connecting to 192.168.202.138@tcp116 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15145.769845] LNetError: Skipped 4 previous similar messages [15145.900694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15145.931570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15146.039482] Lustre: lustre-MDT0000: new disk, initializing [15146.126837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15146.161475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15146.657278] LNet: 4 local NIs in recovery (showing 4): 192.168.202.138@tcp123, 192.168.202.138@tcp124, 192.168.202.138@tcp125, 192.168.202.138@tcp126 [15146.658265] LNetError: 464094:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15146.686618] LNetError: 464094:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15146.707456] LNetError: Protocol error connecting to 192.168.202.138@tcp123 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15146.732347] LNetError: Skipped 8 previous similar messages [15147.682079] LNetError: 464097:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15148.707527] LNetError: Protocol error connecting to 192.168.202.138@tcp123 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15148.718100] LNetError: Skipped 1 previous similar message [15149.728501] LNetError: 464094:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15149.736437] LNetError: 464094:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15151.475809] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15152.802783] LNetError: Protocol error connecting to 192.168.202.138@tcp123 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15152.836245] LNetError: Skipped 1 previous similar message [15153.824721] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15153.832108] LNetError: 464102:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15160.995983] LNetError: Protocol error connecting to 192.168.202.138@tcp123 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15161.004418] LNetError: Skipped 1 previous similar message [15162.018111] LNetError: 464096:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15162.037272] LNetError: 464096:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15164.181754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15164.384475] Lustre: 464092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15164.443318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15164.447786] Lustre: Skipped 1 previous similar message [15164.510275] Lustre: lustre-MDT0001: new disk, initializing [15164.545458] Lustre: 464897:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [15164.552540] Lustre: 464897:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15164.664966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15164.692164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15164.703112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15168.962239] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15171.235051] LNetError: Protocol error connecting to 192.168.202.138@tcp131 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15171.252881] LNetError: Skipped 29 previous similar messages [15173.194605] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [15173.206415] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15178.338032] LNetError: 464098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15178.358864] LNetError: 464098:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 33 previous similar messages [15179.612656] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15185.790040] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15187.554712] LNetError: Protocol error connecting to 192.168.202.138@tcp108 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15187.565654] LNetError: Skipped 17 previous similar messages [15194.041309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15194.473736] Lustre: lustre-OST0000: new disk, initializing [15194.482587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15194.490652] Lustre: 466069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15194.603730] Lustre: 466074:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [15194.612591] Lustre: 466074:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15194.673393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15200.819030] Lustre: 466363:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.138@tcp: can't add NID 192.168.202.138@tcp130: rc = -75 [15200.826174] Lustre: 466363:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15200.889078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15200.907649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15201.093668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15202.114217] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15207.071484] LNet: 7 local NIs in recovery (showing 7): 192.168.202.138@tcp127, 192.168.202.138@tcp129, 192.168.202.138@tcp130, 192.168.202.138@tcp131, 192.168.202.138@tcp133, 192.168.202.138@tcp125, 192.168.202.138@tcp123 [15211.105076] LNetError: 464099:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.138 [15211.117975] LNetError: 464099:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 39 previous similar messages [15214.117645] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15220.324474] LNetError: Protocol error connecting to 192.168.202.138@tcp120 at host 192.168.202.138:988: is it running a compatible version of Lustre? [15220.343356] LNetError: Skipped 35 previous similar messages [15226.853139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15226.854537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15226.865315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15226.873363] LustreError: Skipped 1 previous similar message [15226.903993] Lustre: Skipped 1 previous similar message [15229.514929] Lustre: server umount lustre-OST0000 complete [15236.578209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15236.586989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15236.607100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15236.625744] Lustre: Skipped 1 previous similar message [15242.217193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15242.223710] Lustre: Skipped 3 previous similar messages [15247.328122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15248.355084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15248.580896] Lustre: server umount lustre-MDT0000 complete [15252.534281] LustreError: 464084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031389 with bad export cookie 3901620636012059856 [15252.539722] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15252.546956] LustreError: 464084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15253.099587] Lustre: server umount lustre-MDT0001 complete [15264.723150] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [15268.450408] Key type lgssc unregistered [15268.971244] LNet: 467983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15268.988938] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15268.992374] LNet: Removed LNI 192.168.202.138@tcp [15270.047163] LNet: Removed LNI 192.168.202.138@tcp100 [15270.052065] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15270.068220] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15270.072238] LNet: Removed LNI 192.168.202.138@tcp101 [15271.135341] LNet: Removed LNI 192.168.202.138@tcp102 [15271.156020] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15271.168751] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15271.180558] LNet: Removed LNI 192.168.202.138@tcp103 [15272.224270] LNet: Removed LNI 192.168.202.138@tcp104 [15273.248097] LNet: Removed LNI 192.168.202.138@tcp105 [15273.251514] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15273.266364] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15273.291346] LNet: Removed LNI 192.168.202.138@tcp106 [15274.336829] LNet: Removed LNI 192.168.202.138@tcp107 [15275.359184] LNet: Removed LNI 192.168.202.138@tcp108 [15275.366512] LNet: Removed LNI 192.168.202.138@tcp109 [15276.383242] LNet: Removed LNI 192.168.202.138@tcp110 [15277.407214] LNet: Removed LNI 192.168.202.138@tcp111 [15277.409404] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15277.443749] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [15277.456692] LNet: Removed LNI 192.168.202.138@tcp112 [15278.495610] LNet: Removed LNI 192.168.202.138@tcp113 [15279.519208] LNet: Removed LNI 192.168.202.138@tcp114 [15280.607127] LNet: Removed LNI 192.168.202.138@tcp115 [15281.631424] LNet: Removed LNI 192.168.202.138@tcp116 [15282.655193] LNet: Removed LNI 192.168.202.138@tcp117 [15283.679864] LNet: Removed LNI 192.168.202.138@tcp118 [15284.703201] LNet: Removed LNI 192.168.202.138@tcp119 [15285.727524] LNet: Removed LNI 192.168.202.138@tcp120 [15285.730367] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15285.734223] LNetError: 467983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15285.739881] LNet: Removed LNI 192.168.202.138@tcp121 [15286.751784] LNet: Removed LNI 192.168.202.138@tcp122 [15286.759888] LNet: Removed LNI 192.168.202.138@tcp123 [15287.775148] LNet: Removed LNI 192.168.202.138@tcp124 [15287.779907] LNet: Removed LNI 192.168.202.138@tcp125 [15288.863143] LNet: Removed LNI 192.168.202.138@tcp126 [15288.870107] LNet: Removed LNI 192.168.202.138@tcp127 [15289.887569] LNet: Removed LNI 192.168.202.138@tcp128 [15289.890761] LNet: Removed LNI 192.168.202.138@tcp129 [15289.893215] LNet: Removed LNI 192.168.202.138@tcp130 [15289.899142] LNet: Removed LNI 192.168.202.138@tcp131 [15290.911150] LNet: Removed LNI 192.168.202.138@tcp132 [15291.935437] LNet: Removed LNI 192.168.202.138@tcp133 [15292.959948] LNet: Removed LNI 192.168.202.138@tcp134 [15293.994659] LNet: Removed LNI 192.168.202.138@tcp135 [15295.092170] Key type .llcrypt unregistered [15295.096198] Key type ._llcrypt unregistered [15320.039862] Key type ._llcrypt registered [15320.043968] Key type .llcrypt registered [15320.190997] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [15338.002725] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [15339.501531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15339.592940] alg: No test for adler32 (adler32-zlib) [15340.695241] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [15340.936308] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [15342.616945] Key type lgssc registered [15344.299229] Lustre: Echo OBD driver; http://www.lustre.org/ [15381.794962] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:11:57 (1789031517) [15402.435292] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [15405.314183] Key type lgssc unregistered [15405.532664] LNet: 472068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15405.541216] LNetError: 472068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15405.564202] LNet: Removed LNI 192.168.202.138@tcp [15406.458455] Key type .llcrypt unregistered [15406.460243] Key type ._llcrypt unregistered [15423.530476] Key type ._llcrypt registered [15423.532298] Key type .llcrypt registered [15423.665441] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [15425.265378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15425.304063] alg: No test for adler32 (adler32-zlib) [15426.337655] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [15426.542309] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [15428.199331] Key type lgssc registered [15429.225775] Lustre: Echo OBD driver; http://www.lustre.org/ [15439.235897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15439.265044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15440.539562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15440.560400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15440.623889] Lustre: lustre-MDT0000: new disk, initializing [15440.707079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15440.727897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15446.087739] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15457.321986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15457.442835] Lustre: 473278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15457.491918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15457.497404] Lustre: Skipped 1 previous similar message [15457.556553] Lustre: lustre-MDT0001: new disk, initializing [15457.652394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15457.685871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15457.699355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15461.850711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15470.950897] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15477.190287] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15484.681393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15485.069657] Lustre: lustre-OST0000: new disk, initializing [15485.078490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15485.089874] Lustre: 475249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15485.249822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15493.179821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15493.246245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15493.274769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15493.504791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15506.078944] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15509.743729] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing lnet_if_list [15516.552493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15516.596474] LNet: Added LNI 192.168.202.138@tcp10 [8/256/0/180] [15524.697086] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15524.740678] LNet: Added LNI 192.168.202.138@tcp11 [8/256/0/180] [15529.511953] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15529.529889] LNet: Added LNI 192.168.202.138@tcp12 [8/256/0/180] [15534.740917] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15534.783950] LNet: Added LNI 192.168.202.138@tcp13 [8/256/0/180] [15540.119724] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15540.140654] LNet: Added LNI 192.168.202.138@tcp14 [8/256/0/180] [15545.349497] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15545.377163] LNet: Added LNI 192.168.202.138@tcp15 [8/256/0/180] [15550.631528] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15550.677952] LNet: Added LNI 192.168.202.138@tcp16 [8/256/0/180] [15556.275474] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15556.305369] LNet: Added LNI 192.168.202.138@tcp17 [8/256/0/180] [15561.253382] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15561.279565] LNet: Added LNI 192.168.202.138@tcp18 [8/256/0/180] [15566.634451] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15566.679548] LNet: Added LNI 192.168.202.138@tcp19 [8/256/0/180] [15571.676034] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15571.706825] LNet: Added LNI 192.168.202.138@tcp20 [8/256/0/180] [15577.760779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15577.829870] LNet: Added LNI 192.168.202.138@tcp21 [8/256/0/180] [15584.267387] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15584.301744] LNet: Added LNI 192.168.202.138@tcp22 [8/256/0/180] [15589.722683] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15589.752481] LNet: Added LNI 192.168.202.138@tcp23 [8/256/0/180] [15596.240790] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15596.262200] LNet: Added LNI 192.168.202.138@tcp24 [8/256/0/180] [15602.243125] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15602.278946] LNet: Added LNI 192.168.202.138@tcp25 [8/256/0/180] [15607.931249] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15607.973744] LNet: Added LNI 192.168.202.138@tcp26 [8/256/0/180] [15613.690030] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15613.739029] LNet: Added LNI 192.168.202.138@tcp27 [8/256/0/180] [15620.417464] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15620.444601] LNet: Added LNI 192.168.202.138@tcp28 [8/256/0/180] [15627.166702] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15627.204807] LNet: Added LNI 192.168.202.138@tcp29 [8/256/0/180] [15634.469585] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15634.493942] LNet: Added LNI 192.168.202.138@tcp30 [8/256/0/180] [15642.014539] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15642.037942] LNet: Added LNI 192.168.202.138@tcp31 [8/256/0/180] [15648.964714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15648.989390] LNet: Added LNI 192.168.202.138@tcp32 [8/256/0/180] [15656.613109] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15656.655585] LNet: Added LNI 192.168.202.138@tcp33 [8/256/0/180] [15663.475679] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15663.504769] LNet: Added LNI 192.168.202.138@tcp34 [8/256/0/180] [15671.683784] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15671.725091] LNet: Added LNI 192.168.202.138@tcp35 [8/256/0/180] [15679.280790] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15679.325720] LNet: Added LNI 192.168.202.138@tcp36 [8/256/0/180] [15687.954478] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15687.999306] LNet: Added LNI 192.168.202.138@tcp37 [8/256/0/180] [15697.996244] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15698.055709] LNet: Added LNI 192.168.202.138@tcp38 [8/256/0/180] [15705.780741] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15705.833137] LNet: Added LNI 192.168.202.138@tcp39 [8/256/0/180] [15713.743616] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15713.779795] LNet: Added LNI 192.168.202.138@tcp40 [8/256/0/180] [15721.989113] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15722.020449] LNet: Added LNI 192.168.202.138@tcp41 [8/256/0/180] [15730.500392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15730.549549] LNet: Added LNI 192.168.202.138@tcp42 [8/256/0/180] [15739.341255] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15739.412497] LNet: Added LNI 192.168.202.138@tcp43 [8/256/0/180] [15750.839862] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15750.867710] LNet: Added LNI 192.168.202.138@tcp44 [8/256/0/180] [15760.939236] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15760.958672] LNet: Added LNI 192.168.202.138@tcp45 [8/256/0/180] [15769.624352] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15769.654697] LNet: Added LNI 192.168.202.138@tcp46 [8/256/0/180] [15778.128739] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15778.163831] LNet: Added LNI 192.168.202.138@tcp47 [8/256/0/180] [15786.892474] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15786.918767] LNet: Added LNI 192.168.202.138@tcp48 [8/256/0/180] [15794.415391] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15794.444494] LNet: Added LNI 192.168.202.138@tcp49 [8/256/0/180] [15803.086435] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15803.116599] LNet: Added LNI 192.168.202.138@tcp50 [8/256/0/180] [15826.413870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15826.414569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15826.415781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15826.436080] Lustre: Skipped 1 previous similar message [15828.510409] Lustre: server umount lustre-MDT0000 complete [15832.796311] LustreError: 473269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031969 with bad export cookie 3037081200719559268 [15832.799654] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15832.814943] LustreError: 473269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15833.164919] Lustre: server umount lustre-MDT0001 complete [15853.025246] Lustre: 472548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789031973/real 1789031973] req@00000000e57edb9d x1875935560384000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789031989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15853.053185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15853.060130] Lustre: Skipped 1 previous similar message [15853.669953] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [15854.117526] Lustre: 472548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789031974/real 1789031974] req@00000000a3b3164c x1875935560384128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789031990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15858.155085] Lustre: 472550:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789031978/real 1789031978] req@000000001c863e79 x1875935560384256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789031994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15864.030877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15864.287956] Lustre: 472549:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789031984/real 1789031984] req@00000000934180d2 x1875935560384640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789032000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15864.348022] Lustre: 472549:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15868.511437] Lustre: 472549:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789031989/real 1789031989] req@00000000b0221f5d x1875935560384768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789032005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15877.689197] LustreError: 472547:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000024cbb864 x1875935560385408/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15878.092668] LustreError: 475241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15878.120336] LustreError: 475241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15878.216657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15882.211726] LustreError: 475242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15883.232344] LustreError: 475241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15883.240072] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15884.952289] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15888.358786] LustreError: 475242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15893.478588] LustreError: 475241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15894.533903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15894.906646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15897.964205] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15899.414138] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [15906.386785] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15912.592935] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15935.968874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15935.987517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15936.017224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15936.027056] Lustre: Skipped 2 previous similar messages [15939.048940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15939.049847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15944.162112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15944.173827] Lustre: Skipped 2 previous similar messages [15949.289794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15949.293153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [15949.295673] Lustre: Skipped 1 previous similar message [15949.456843] Lustre: server umount lustre-OST0000 complete [15954.421732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15954.447921] Lustre: Skipped 1 previous similar message [15959.416735] Lustre: server umount lustre-MDT0000 complete [15959.520407] LustreError: 499496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15959.541520] LustreError: 499496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15963.293453] LustreError: 475247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789032099 with bad export cookie 3037081200719561900 [15963.300391] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15963.307862] LustreError: 475247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15963.586916] Lustre: server umount lustre-MDT0001 complete [15974.005733] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [15976.743460] Key type lgssc unregistered [15977.176219] LNet: 502295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15977.182778] LNetError: 502295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15978.207220] LNet: Removed LNI 192.168.202.138@tcp [15978.210378] LNetError: 502295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15978.221664] LNet: Removed LNI 192.168.202.138@tcp10 [15978.227475] LNet: Removed LNI 192.168.202.138@tcp11 [15978.232899] LNet: Removed LNI 192.168.202.138@tcp12 [15978.239075] LNet: Removed LNI 192.168.202.138@tcp13 [15978.242974] LNet: Removed LNI 192.168.202.138@tcp14 [15978.245670] LNet: Removed LNI 192.168.202.138@tcp15 [15978.258045] LNet: Removed LNI 192.168.202.138@tcp16 [15978.267406] LNet: Removed LNI 192.168.202.138@tcp17 [15978.272517] LNet: Removed LNI 192.168.202.138@tcp18 [15978.281590] LNet: Removed LNI 192.168.202.138@tcp19 [15978.283317] LNet: Removed LNI 192.168.202.138@tcp20 [15978.284871] LNet: Removed LNI 192.168.202.138@tcp21 [15978.294567] LNet: Removed LNI 192.168.202.138@tcp22 [15978.298479] LNet: Removed LNI 192.168.202.138@tcp23 [15978.303863] LNet: Removed LNI 192.168.202.138@tcp24 [15978.314456] LNet: Removed LNI 192.168.202.138@tcp25 [15978.317568] LNet: Removed LNI 192.168.202.138@tcp26 [15978.323435] LNet: Removed LNI 192.168.202.138@tcp27 [15978.331585] LNet: Removed LNI 192.168.202.138@tcp28 [15978.336464] LNet: Removed LNI 192.168.202.138@tcp29 [15978.340471] LNet: Removed LNI 192.168.202.138@tcp30 [15978.344412] LNet: Removed LNI 192.168.202.138@tcp31 [15978.353544] LNet: Removed LNI 192.168.202.138@tcp32 [15978.361302] LNet: Removed LNI 192.168.202.138@tcp33 [15978.365213] LNet: Removed LNI 192.168.202.138@tcp34 [15978.369468] LNet: Removed LNI 192.168.202.138@tcp35 [15978.377571] LNet: Removed LNI 192.168.202.138@tcp36 [15978.392562] LNet: Removed LNI 192.168.202.138@tcp37 [15978.397738] LNet: Removed LNI 192.168.202.138@tcp38 [15978.403190] LNet: Removed LNI 192.168.202.138@tcp39 [15978.407709] LNet: Removed LNI 192.168.202.138@tcp40 [15978.420262] LNet: Removed LNI 192.168.202.138@tcp41 [15978.423491] LNet: Removed LNI 192.168.202.138@tcp42 [15978.438334] LNet: Removed LNI 192.168.202.138@tcp43 [15978.445992] LNet: Removed LNI 192.168.202.138@tcp44 [15978.450116] LNet: Removed LNI 192.168.202.138@tcp45 [15978.453304] LNet: Removed LNI 192.168.202.138@tcp46 [15978.475274] LNet: Removed LNI 192.168.202.138@tcp47 [15978.484329] LNet: Removed LNI 192.168.202.138@tcp48 [15978.496462] LNet: Removed LNI 192.168.202.138@tcp49 [15978.513083] LNet: Removed LNI 192.168.202.138@tcp50 [15979.240106] Key type .llcrypt unregistered [15979.242779] Key type ._llcrypt unregistered [15996.735785] Key type ._llcrypt registered [15996.737984] Key type .llcrypt registered [15996.819752] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [16009.120303] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [16009.962395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16010.036700] alg: No test for adler32 (adler32-zlib) [16011.061542] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [16011.273577] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [16012.936899] Key type lgssc registered [16013.867633] Lustre: Echo OBD driver; http://www.lustre.org/ [16045.620327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16045.649856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16046.857606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16046.893788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16046.963272] Lustre: lustre-MDT0000: new disk, initializing [16047.030089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16047.052104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16051.329522] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16061.906488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16062.076382] Lustre: 505467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16062.130159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16062.134713] Lustre: Skipped 1 previous similar message [16062.206713] Lustre: lustre-MDT0001: new disk, initializing [16062.311417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16062.349245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16062.361473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16066.620807] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16076.111759] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16082.784839] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16091.223989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16091.466210] Lustre: *** cfs_fail_loc=90f, val=0*** [16091.603753] Lustre: lustre-OST0000: new disk, initializing [16091.608802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16091.616332] Lustre: 507484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16091.741799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16097.915031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16097.938636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16098.146261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16099.231970] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16110.006937] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16123.876359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16123.883156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16123.888614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16123.909610] Lustre: Skipped 1 previous similar message [16126.034580] Lustre: server umount lustre-OST0000 complete [16134.113839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16134.116483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16134.122433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16134.122442] Lustre: Skipped 1 previous similar message [16134.135546] Lustre: Skipped 1 previous similar message [16134.173488] LustreError: Skipped 1 previous similar message [16139.232475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16139.237966] Lustre: Skipped 2 previous similar messages [16144.351295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16144.354189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16144.383946] Lustre: Skipped 1 previous similar message [16144.697292] Lustre: server umount lustre-MDT0000 complete [16148.847824] LustreError: 506271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789032285 with bad export cookie 11079212111497565066 [16148.862242] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16148.874961] LustreError: 506271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16149.693282] Lustre: server umount lustre-MDT0001 complete [16162.411543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [16165.254092] Key type lgssc unregistered [16165.603714] LNet: 509391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16165.612450] LNetError: 509391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16165.630777] LNet: Removed LNI 192.168.202.138@tcp [16166.829095] Key type .llcrypt unregistered [16166.832536] Key type ._llcrypt unregistered [16190.481336] Key type ._llcrypt registered [16190.483277] Key type .llcrypt registered [16190.582810] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [16192.067177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16192.112089] alg: No test for adler32 (adler32-zlib) [16193.198826] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [16193.444735] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [16195.167746] Key type lgssc registered [16196.402022] Lustre: Echo OBD driver; http://www.lustre.org/ [16205.110252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16205.136995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16206.861938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16211.257947] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16217.071826] LustreError: 510661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16217.087811] LustreError: 510661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16222.030741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16222.176823] LustreError: 510660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16222.490796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16227.813608] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16235.970985] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16242.236368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16249.910722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16250.311774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16257.893565] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16266.866984] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16276.581534] loop0: detected capacity change from 0 to 204800000 [16276.800284] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16276.809976] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16276.820676] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16282.528818] loop0: detected capacity change from 0 to 204800000 [16282.677986] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16282.691601] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16282.699686] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16289.157818] loop0: detected capacity change from 0 to 204800000 [16296.673194] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16296.893683] Lustre: fs15246-OST0000: new disk, initializing [16296.899264] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16296.908447] Lustre: 514257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16296.998376] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16304.623400] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16314.993808] loop1: detected capacity change from 0 to 204800000 [16321.116808] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16321.262995] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16321.300733] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16321.443630] Lustre: fs15246-MDT0000: new disk, initializing [16321.685793] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16321.741725] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16321.885865] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16322.918014] Lustre: 515379:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000dc62f2de x1875936363462016/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789032475 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16326.809447] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16332.008100] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16332.016535] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16332.039403] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16354.790419] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16354.812875] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16356.780751] Lustre: server umount fs15246-MDT0000 complete [16359.920378] LustreError: 510655:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16365.030126] LustreError: 510657:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16367.180563] Lustre: server umount fs15246-OST0000 complete [16384.512284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16384.522357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16384.547981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16385.505024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16387.608637] Lustre: server umount lustre-OST0000 complete [16395.749655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16395.752753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16395.784389] Lustre: Skipped 2 previous similar messages [16395.797949] Lustre: Skipped 2 previous similar messages [16398.733371] Lustre: server umount lustre-MDT0000 complete [16400.869093] LustreError: 511642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16400.883560] LustreError: 511642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16402.630056] LustreError: 512503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789032539 with bad export cookie 5513661445989707248 [16402.636367] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16402.640497] LustreError: 512503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16403.177586] Lustre: server umount lustre-MDT0001 complete [16414.225167] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [16416.376537] Key type lgssc unregistered [16416.669397] LNet: 518475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16416.678777] LNetError: 518475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16416.694359] LNet: Removed LNI 192.168.202.138@tcp [16417.734526] Key type .llcrypt unregistered [16417.740324] Key type ._llcrypt unregistered [16437.155157] Key type ._llcrypt registered [16437.157596] Key type .llcrypt registered [16437.328730] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [16438.850872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16438.874482] alg: No test for adler32 (adler32-zlib) [16439.941795] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [16440.311084] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [16441.991333] Key type lgssc registered [16443.383828] Lustre: Echo OBD driver; http://www.lustre.org/ [16484.443950] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [16494.810986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [16531.904689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16531.933916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16533.210554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16533.241588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16533.322633] Lustre: lustre-MDT0000: new disk, initializing [16533.439865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16533.457243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16537.546799] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16549.183256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16549.377528] Lustre: 522939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16549.455037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16549.459460] Lustre: Skipped 1 previous similar message [16549.532285] Lustre: lustre-MDT0001: new disk, initializing [16549.663181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16549.723452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16549.745858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16553.610549] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16564.108600] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16571.864467] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16579.258152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16579.530259] Lustre: lustre-OST0000: new disk, initializing [16579.534726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16579.544604] Lustre: 524907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16579.662986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16586.381931] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [16587.854809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16587.865634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16588.073511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16596.070798] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16601.220131] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16601.444511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16606.765962] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16606.966756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16608.737377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16608.741264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16608.744841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16608.745696] LustreError: Skipped 1 previous similar message [16608.765379] Lustre: Skipped 1 previous similar message [16613.866120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16613.868890] Lustre: Skipped 2 previous similar messages [16615.141791] Lustre: server umount lustre-OST0000 complete [16618.986265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16618.992249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16619.008350] Lustre: Skipped 1 previous similar message [16619.022645] Lustre: Skipped 1 previous similar message [16624.097087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16624.102672] Lustre: Skipped 1 previous similar message [16629.216823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16629.228546] Lustre: Skipped 1 previous similar message [16633.312485] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16633.585285] Lustre: server umount lustre-MDT0000 complete [16637.842213] LustreError: 524906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789032774 with bad export cookie 16261910256327504304 [16637.847884] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16637.857698] LustreError: 524906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16638.347705] Lustre: server umount lustre-MDT0001 complete [16667.829372] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [16679.084969] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [16719.316984] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16442 sec ========= 05:34:13 (1789032853) [16721.614199] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:34:16 (1789032856) === [16725.991922] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:34:21 (1789032861) === [16766.962843] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [16771.019976] Key type lgssc unregistered [16771.434615] LNet: 531556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16771.445196] LNetError: 531556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16771.471751] LNet: Removed LNI 192.168.202.138@tcp [16772.567555] Key type .llcrypt unregistered [16772.569543] Key type ._llcrypt unregistered