[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 529705543 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007923] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.021335] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026303] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031129] *** VALIDATE tmpfs *** [ 0.032557] *** VALIDATE proc *** [ 0.034121] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.037146] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039054] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042031] Spectre V2 : User space: Vulnerable [ 0.043000] Speculative Store Bypass: Vulnerable [ 0.045452] debug: unmapping init [mem 0xffffffffbbe59000-0xffffffffbbe60fff] [ 0.047198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.049996] ... bit width: 48 [ 0.050022] ... generic registers: 4 [ 0.050971] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.052973] ... event mask: 000000070000000f [ 0.053329] rcu: Hierarchical SRCU implementation. [ 0.055612] smp: Bringing up secondary CPUs ... [ 0.056662] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060247] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142301] node 0 deferred pages initialised in 77ms [ 0.145116] devtmpfs: initialized [ 0.146272] x86/mm: Memory block size: 128MB [ 0.149972] gcov: version magic: 0x41383552 [ 0.152323] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158388] pinctrl core: initialized pinctrl subsystem [ 0.160221] [ 0.160676] ************************************************************* [ 0.162017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ** ** [ 0.166010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168014] ** ** [ 0.169011] ** This means that this kernel is built to expose internal ** [ 0.171017] ** IOMMU data structures, which may compromise security on ** [ 0.174015] ** your system. ** [ 0.177017] ** ** [ 0.179012] ** If you see this message and you are not debugging the ** [ 0.181016] ** kernel, report this immediately to your vendor! ** [ 0.184016] ** ** [ 0.186014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188016] ************************************************************* [ 0.191670] NET: Registered protocol family 16 [ 0.193440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201123] cpuidle: using governor menu [ 0.203187] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204654] PCI: Using configuration type 1 for base access [ 0.206146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222069] cryptd: max_cpu_qlen set to 1000 [ 0.226290] ACPI: Added _OSI(Module Device) [ 0.228021] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236301] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243575] ACPI: Interpreter enabled [ 0.245065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246018] ACPI: Using IOAPIC for interrupt routing [ 0.248108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252439] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277550] acpiphp: Slot [2] registered [ 0.279144] acpiphp: Slot [5] registered [ 0.280166] acpiphp: Slot [6] registered [ 0.282146] acpiphp: Slot [7] registered [ 0.284105] acpiphp: Slot [8] registered [ 0.285140] acpiphp: Slot [9] registered [ 0.287147] acpiphp: Slot [10] registered [ 0.288140] acpiphp: Slot [3] registered [ 0.290149] acpiphp: Slot [4] registered [ 0.291129] acpiphp: Slot [11] registered [ 0.293101] acpiphp: Slot [12] registered [ 0.294106] acpiphp: Slot [13] registered [ 0.296162] acpiphp: Slot [14] registered [ 0.297121] acpiphp: Slot [15] registered [ 0.299122] acpiphp: Slot [16] registered [ 0.300076] acpiphp: Slot [17] registered [ 0.301097] acpiphp: Slot [18] registered [ 0.302074] acpiphp: Slot [19] registered [ 0.304107] acpiphp: Slot [20] registered [ 0.305119] acpiphp: Slot [21] registered [ 0.307104] acpiphp: Slot [22] registered [ 0.308115] acpiphp: Slot [23] registered [ 0.309100] acpiphp: Slot [24] registered [ 0.311124] acpiphp: Slot [25] registered [ 0.312115] acpiphp: Slot [26] registered [ 0.314135] acpiphp: Slot [27] registered [ 0.315108] acpiphp: Slot [28] registered [ 0.316116] acpiphp: Slot [29] registered [ 0.318126] acpiphp: Slot [30] registered [ 0.319093] acpiphp: Slot [31] registered [ 0.321064] PCI host bridge to bus 0000:00 [ 0.322025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339960] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343303] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352772] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359120] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370526] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379902] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.410911] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431033] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469344] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.482021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.516022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.528835] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554037] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.576028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.586606] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.598033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.611028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.638025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.649103] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.660045] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.685027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.702659] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.716017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.755022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.771357] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.773590] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.776479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.779479] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782305] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.787116] iommu: Default domain type: Passthrough [ 0.789522] SCSI subsystem initialized [ 0.792249] ACPI: bus type USB registered [ 0.794182] usbcore: registered new interface driver usbfs [ 0.796124] usbcore: registered new interface driver hub [ 0.798133] usbcore: registered new device driver usb [ 0.800255] pps_core: LinuxPPS API ver. 1 registered [ 0.802022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805087] PTP clock support registered [ 0.807169] EDAC MC: Ver: 3.0.0 [ 0.809145] PCI: Using ACPI for IRQ routing [ 0.810873] NetLabel: Initializing [ 0.812010] NetLabel: domain hash size = 128 [ 0.813011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.815079] NetLabel: unlabeled traffic allowed by default [ 0.816295] vgaarb: loaded [ 0.817248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.818071] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.827097] clocksource: Switched to clocksource kvm-clock [ 0.915918] VFS: Disk quotas dquot_6.6.0 [ 0.916904] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.918452] *** VALIDATE ramfs *** [ 0.919306] *** VALIDATE hugetlbfs *** [ 0.921577] pnp: PnP ACPI init [ 0.924152] pnp: PnP ACPI: found 6 devices [ 0.940526] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.943909] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945969] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.948028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.950350] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.952724] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955597] NET: Registered protocol family 2 [ 0.958039] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.962556] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.965933] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.971107] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.974644] TCP: Hash tables configured (established 65536 bind 65536) [ 0.977534] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.980770] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983577] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986604] NET: Registered protocol family 1 [ 0.988795] RPC: Registered named UNIX socket transport module. [ 0.991800] RPC: Registered udp transport module. [ 0.993479] RPC: Registered tcp transport module. [ 0.995102] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.996696] NET: Registered protocol family 44 [ 0.998323] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.000206] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.002263] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.004064] PCI: CLS 0 bytes, default 64 [ 1.006410] Unpacking initramfs... [ 2.386134] debug: unmapping init [mem 0xffff8cff3cc54000-0xffff8cff3ffbffff] [ 2.390091] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.392611] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.395939] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.879241] Initialise system trusted keyrings [ 2.880754] Key type blacklist registered [ 2.882731] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.890341] zbud: loaded [ 2.892842] *** VALIDATE nfs *** [ 2.894180] *** VALIDATE nfs4 *** [ 2.895770] pstore: using deflate compression [ 2.899164] Platform Keyring initialized [ 2.976333] NET: Registered protocol family 38 [ 2.977606] Key type asymmetric registered [ 2.978864] Asymmetric key parser 'x509' registered [ 2.980145] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.982227] io scheduler mq-deadline registered [ 2.983338] io scheduler kyber registered [ 2.984708] io scheduler bfq registered [ 2.985881] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.987851] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.989709] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.991518] ACPI: Power Button [PWRF] [ 2.995243] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.002114] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.015682] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.024180] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.043165] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070076] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.100741] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.104671] Non-volatile memory driver v1.3 [ 3.105684] Linux agpgart interface v0.103 [ 3.132712] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.134349] vda: detected capacity change from 0 to 74698752 [ 3.145486] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.147273] vdb: detected capacity change from 0 to 1073741824 [ 3.163570] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165964] vdc: detected capacity change from 0 to 2621440000 [ 3.179185] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.180890] vdd: detected capacity change from 0 to 2621440000 [ 3.193682] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.196775] vde: detected capacity change from 0 to 4294967296 [ 3.216784] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.219920] vdf: detected capacity change from 0 to 4294967296 [ 3.234743] libphy: Fixed MDIO Bus: probed [ 3.244507] usbcore: registered new interface driver usbserial_generic [ 3.247100] usbserial: USB Serial support registered for generic [ 3.249301] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.253401] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.255084] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.257209] mousedev: PS/2 mouse device common for all mice [ 3.261410] rtc_cmos 00:05: RTC can wake from S4 [ 3.262993] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.266095] rtc_cmos 00:05: registered as rtc0 [ 3.268571] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.272089] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.274824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.276784] intel_pstate: CPU model not supported [ 3.282756] hid: raw HID events driver (C) Jiri Kosina [ 3.284735] usbcore: registered new interface driver usbhid [ 3.286791] usbhid: USB HID core driver [ 3.288342] drop_monitor: Initializing network drop monitor service [ 3.290457] Initializing XFRM netlink socket [ 3.292607] NET: Registered protocol family 10 [ 3.295779] Segment Routing with IPv6 [ 3.297287] NET: Registered protocol family 17 [ 3.299132] mpls_gso: MPLS GSO support [ 3.304460] RAS: Correctable Errors collector initialized. [ 3.306557] AVX version of gcm_enc/dec engaged. [ 3.308150] AES CTR mode by8 optimization enabled [ 3.372798] sched_clock: Marking stable (3372775286, 0)->(4266852259, -894076973) [ 3.376344] registered taskstats version 1 [ 3.377923] Loading compiled-in X.509 certificates [ 3.379883] zswap: loaded using pool lzo/zbud [ 3.397546] Key type big_key registered [ 3.407124] Key type encrypted registered [ 3.408619] ima: No TPM chip found, activating TPM-bypass! [ 3.412041] ima: Allocated hash algorithm: sha1 [ 3.413525] ima: No architecture policies found [ 3.415156] evm: Initialising EVM extended attributes: [ 3.417187] evm: security.selinux [ 3.418353] evm: security.ima [ 3.419422] evm: security.capability [ 3.420861] evm: HMAC attrs: 0x1 [ 3.423638] rtc_cmos 00:05: setting system clock to 2026-09-03 13:23:43 UTC (1788441823) [ 3.429996] debug: unmapping init [mem 0xffffffffbce03000-0xffffffffbcffffff] [ 3.433298] debug: unmapping init [mem 0xffffffffbbb82000-0xffffffffbbe58fff] [ 3.446130] Write protecting the kernel read-only data: 28672k [ 3.449515] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.452401] debug: unmapping init [mem 0xffffffffbab14000-0xffffffffbabfffff] [ 3.488878] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.496975] systemd[1]: Detected virtualization kvm. [ 3.498756] systemd[1]: Detected architecture x86-64. [ 3.500443] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.527573] systemd[1]: No hostname configured. [ 3.529358] systemd[1]: Set hostname to . [ 3.531577] random: systemd: uninitialized urandom read (16 bytes read) [ 3.534202] systemd[1]: Initializing machine ID from random generator. [ 3.560959] random: ln: uninitialized urandom read (6 bytes read) [ 3.657752] random: systemd: uninitialized urandom read (16 bytes read) [ 3.659932] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.664090] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.667333] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.341474] device-mapper: uevent: version 1.0.3 [ 4.344959] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.117384] virtio_net virtio0 ens2: renamed from eth0 [ 5.163928] random: fast init done [ 5.194700] scsi host0: ata_piix [ 5.273186] scsi host1: ata_piix [ 5.274896] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.277394] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.925745] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.927243] random: crng init done [ 9.929268] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.353548] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.405276] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.612533] SELinux: Disabled at runtime. [ 11.677501] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.686380] systemd[1]: Detected virtualization kvm. [ 11.688139] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.154125] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.158168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.162895] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.169379] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.172622] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.180694] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.185324] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ 12.349217] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all De[ 12.569778] squashfs: version 4.0 (2009/01/31) Phillip Lougher vices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.827612] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.879125] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.980870] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.989874] EDAC sbridge: Ver: 1.1.2 [ 14.188528] Key type dns_resolver registered [ 14.493579] NFS: Registering the id_resolver key type [ 14.495570] Key type id_resolver registered [ 14.497181] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 40.740271] libcfs: loading out-of-tree module taints kernel. [ 40.763070] Key type ._llcrypt registered [ 40.764784] Key type .llcrypt registered [ 40.820638] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 60.493906] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 63.052709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.076573] alg: No test for adler32 (adler32-zlib) [ 64.800490] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 65.831416] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 67.583546] Key type lgssc registered [ 69.792284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.073866] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.516308] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 152.491179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.526452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.847348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.899084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.018330] Lustre: lustre-MDT0000: new disk, initializing [ 154.120840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.145120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.835901] hrtimer: interrupt took 6879545 ns [ 159.238727] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.159995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.328175] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.382706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.392140] Lustre: Skipped 1 previous similar message [ 174.515499] Lustre: lustre-MDT0001: new disk, initializing [ 174.622837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.661949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.664838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.124800] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.808264] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.482865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.758380] Lustre: lustre-OST0000: new disk, initializing [ 193.763836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.773984] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.885040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.023488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.036155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.127742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.172675] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.831620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.986477] Lustre: lustre-OST0001: new disk, initializing [ 215.990356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.999695] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.098456] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.426350] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.285985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.302147] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.360203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 235.044677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.048953] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.915279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 259.576813] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 265.147635] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 268.267114] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 270.173385] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 271.658268] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:28:10 EDT 2026 [ 288.028701] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 289.667720] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.684729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 307.697528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.713389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.243945] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.249185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.266294] Lustre: Skipped 3 previous similar messages [ 311.820934] Lustre: server umount lustre-MDT0000 complete [ 315.712924] LustreError: 12257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442135 with bad export cookie 8218948413531189456 [ 315.730754] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.740224] LustreError: 12257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.484283] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.484816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.485489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.515092] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 320.558852] Lustre: Skipped 3 previous similar messages [ 325.600465] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.602730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.623587] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 325.664391] Lustre: Skipped 2 previous similar messages [ 326.641657] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.666653] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 330.207464] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.552059] Lustre: server umount lustre-MDT0001 complete [ 335.318243] Lustre: server umount lustre-OST0000 complete [ 340.536707] Lustre: server umount lustre-OST0001 complete [ 347.104212] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 358.220537] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 396.483588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.712753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.745408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.818267] Lustre: lustre-MDT0000: new disk, initializing [ 396.969469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.019287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.861015] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 413.823877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.022572] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 414.122846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 414.128073] Lustre: Skipped 1 previous similar message [ 414.217986] Lustre: lustre-MDT0001: new disk, initializing [ 414.357692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.396277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.416968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.971710] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 429.241537] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.114201] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.578711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.982760] Lustre: lustre-OST0000: new disk, initializing [ 442.987560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.993335] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.105204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.032403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.042811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.204118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.931383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 459.606358] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.581250] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.845780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.648748] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.870441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.110735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.112688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.125156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.140396] Lustre: Skipped 3 previous similar messages [ 477.982403] Lustre: server umount lustre-OST0000 complete [ 485.348454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.358698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.362380] Lustre: Skipped 2 previous similar messages [ 485.375516] Lustre: Skipped 1 previous similar message [ 488.023917] Lustre: server umount lustre-MDT0000 complete [ 490.465413] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.478849] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 491.341114] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442311 with bad export cookie 8218948413531190982 [ 491.341976] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.347381] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 491.762299] Lustre: server umount lustre-MDT0001 complete [ 496.077920] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:31:54 (1788442314) [ 498.478108] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 500.335553] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:31:59 (1788442319) [ 507.202427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.812307] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.957800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.929235] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 517.093684] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.117958] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 522.613711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.881119] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 536.751455] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 543.598633] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.170798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 558.921950] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 567.001782] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 577.507371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.511858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 577.538143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.543345] LustreError: Skipped 1 previous similar message [ 577.560633] Lustre: Skipped 1 previous similar message [ 579.877805] Lustre: server umount lustre-OST0000 complete [ 584.685889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 590.002746] Lustre: server umount lustre-MDT0000 complete [ 592.867564] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.900783] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 593.559976] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442413 with bad export cookie 8218948413531191871 [ 593.562739] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.571590] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 593.919265] Lustre: server umount lustre-MDT0001 complete [ 605.135711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 607.950759] Key type lgssc unregistered [ 608.360808] LNet: 22989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.373721] LNetError: 22989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.396627] LNet: Removed LNI 192.168.201.112@tcp [ 609.451300] Key type .llcrypt unregistered [ 609.452518] Key type ._llcrypt unregistered [ 632.171249] Key type ._llcrypt registered [ 632.175276] Key type .llcrypt registered [ 632.279491] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 633.361685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.466610] alg: No test for adler32 (adler32-zlib) [ 634.635693] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 634.927386] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 636.575722] Key type lgssc registered [ 638.316889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 647.050809] loop: module loaded [ 647.166067] loop0: detected capacity change from 0 to 204800000 [ 647.388161] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.407344] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.780271] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.472612] loop0: detected capacity change from 0 to 204800000 [ 653.658883] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.675708] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.685733] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.275578] loop0: detected capacity change from 0 to 204800000 [ 664.757807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 664.798936] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.042872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 666.100525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 666.191633] Lustre: lustre-MDT0000: new disk, initializing [ 666.305073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.331362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 670.790559] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 678.967267] loop1: detected capacity change from 0 to 204800000 [ 684.479227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 684.763668] Lustre: lustre-OST07c6: new disk, initializing [ 684.768182] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 684.772666] Lustre: Skipped 1 previous similar message [ 684.780338] Lustre: 26191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 684.850757] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 691.790702] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 693.398718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 693.406097] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 693.437422] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 697.299930] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 697.305185] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 697.309250] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 697.313841] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 697.318648] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 697.323416] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.290791] Lustre: 26763:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 714.297494] Lustre: 26763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.308643] Lustre: 26763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.316060] Lustre: 26763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 714.338447] Lustre: 26763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 714.348154] Lustre: 26763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.222175] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 719.234625] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.245661] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 719.263062] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.275559] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.292316] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.301056] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 719.305918] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.316269] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.331801] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.345212] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.354625] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.551222] Lustre: 26763:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 722.559761] Lustre: 26763:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.567223] Lustre: 26763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.585450] Lustre: 26763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.595994] Lustre: 26763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.616654] Lustre: 26763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.625421] Lustre: 26763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 722.634258] Lustre: 26763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.644769] Lustre: 26763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.654642] Lustre: 26763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.666491] Lustre: 26763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.683162] Lustre: 26763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.322423] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 727.347697] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.365432] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.378326] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.397409] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.412702] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.420599] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 727.425987] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.431310] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.436216] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.441641] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.446476] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.057794] Lustre: 26763:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 737.074552] Lustre: 26763:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 737.090574] Lustre: 26763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 737.098305] Lustre: 26763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.109776] Lustre: 26763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 737.122698] Lustre: 26763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.135406] Lustre: 26763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 737.151596] Lustre: 26763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.168842] Lustre: 26763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 737.178719] Lustre: 26763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.186598] Lustre: 26763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 737.191478] Lustre: 26763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 785.381207] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.394204] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 790.508838] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 791.290233] Lustre: server umount lustre-OST07c6 complete [ 801.000560] Lustre: server umount lustre-MDT0000 complete [ 817.138937] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 819.609652] Key type lgssc unregistered [ 819.873512] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.880292] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.909730] LNet: Removed LNI 192.168.201.112@tcp [ 820.792253] Key type .llcrypt unregistered [ 820.794714] Key type ._llcrypt unregistered [ 837.304326] Key type ._llcrypt registered [ 837.305798] Key type .llcrypt registered [ 837.422936] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 838.356203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.390763] alg: No test for adler32 (adler32-zlib) [ 839.473283] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 839.779108] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 841.415395] Key type lgssc registered [ 842.795878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.205776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.241890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.738937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.234291] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 862.186294] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.238575] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 867.300133] LustreError: 30000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.738064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.595990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.832649] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 886.042937] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.892930] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 899.920664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.430473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.005441] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 914.041986] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 922.960084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 943.587171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.602069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.608336] Lustre: Skipped 1 previous similar message [ 943.631679] Lustre: Skipped 1 previous similar message [ 946.048851] Lustre: server umount lustre-OST0000 complete [ 953.824844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.827423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.857619] Lustre: Skipped 1 previous similar message [ 953.884542] Lustre: Skipped 1 previous similar message [ 956.284463] Lustre: server umount lustre-MDT0000 complete [ 958.945166] LustreError: 29997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.957043] LustreError: 29997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 960.580416] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442780 with bad export cookie 6901764909771748028 [ 960.591802] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.599227] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 961.095857] Lustre: server umount lustre-MDT0001 complete [ 976.982521] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 982.021403] Key type lgssc unregistered [ 982.395910] LNet: 33764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.402289] LNetError: 33764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.436366] LNet: Removed LNI 192.168.201.112@tcp [ 983.489567] Key type .llcrypt unregistered [ 983.493764] Key type ._llcrypt unregistered [ 1005.535104] Key type ._llcrypt registered [ 1005.536774] Key type .llcrypt registered [ 1005.738829] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1007.995764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.023055] alg: No test for adler32 (adler32-zlib) [ 1009.260594] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1009.740543] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1011.431177] Key type lgssc registered [ 1012.899143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.572929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.614204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.125716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.005092] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1033.707136] LustreError: 35035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.731045] LustreError: 35035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1037.450936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.900966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1043.175712] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1050.443862] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1056.097756] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1062.766912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.312527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.474981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1070.621287] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1079.305249] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.098271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.105811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.112136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.132782] Lustre: Skipped 1 previous similar message [ 1098.220928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.233570] Lustre: Skipped 2 previous similar messages [ 1098.912265] Lustre: server umount lustre-OST0000 complete [ 1103.329875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.340091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.344336] Lustre: Skipped 1 previous similar message [ 1103.361049] Lustre: Skipped 1 previous similar message [ 1108.448717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.462136] Lustre: Skipped 1 previous similar message [ 1113.585367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.596086] Lustre: Skipped 1 previous similar message [ 1117.152471] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1117.473317] Lustre: server umount lustre-MDT0000 complete [ 1118.688057] LustreError: 35036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.702237] LustreError: 35036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1121.180256] LustreError: 36877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442941 with bad export cookie 3526180422796293434 [ 1121.187105] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.188972] LustreError: 36877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1121.746548] Lustre: server umount lustre-MDT0001 complete [ 1133.214122] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1136.662272] Key type lgssc unregistered [ 1136.983491] LNet: 38601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.987807] LNetError: 38601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.025432] LNet: Removed LNI 192.168.201.112@tcp [ 1138.997366] Key type .llcrypt unregistered [ 1139.000605] Key type ._llcrypt unregistered [ 1165.016422] Key type ._llcrypt registered [ 1165.018304] Key type .llcrypt registered [ 1165.202388] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1167.282425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.456874] alg: No test for adler32 (adler32-zlib) [ 1168.667814] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1168.980394] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1170.711867] Key type lgssc registered [ 1172.618785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.705865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1182.749933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.556623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.286590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1193.956533] LustreError: 40019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.978981] LustreError: 40019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1198.289353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.738774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1204.296713] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1212.629888] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1219.742749] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1227.233860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1227.559623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.620110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1234.637746] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1241.301383] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1252.963069] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1253.344780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.357943] Lustre: Skipped 1 previous similar message [ 1256.859786] Lustre: server umount lustre-MDT0000 complete [ 1258.048305] LustreError: 41859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.825338] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443080 with bad export cookie 12143826248398981782 [ 1260.844298] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.848439] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1263.159718] LustreError: 40015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.185445] LustreError: 40015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1263.205151] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 1263.211670] Lustre: Skipped 3 previous similar messages [ 1263.584743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.601713] Lustre: Skipped 1 previous similar message [ 1265.648517] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.649579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1265.671603] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1265.679235] Lustre: Skipped 1 previous similar message [ 1268.292431] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 1270.753825] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.765302] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1273.406321] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 1273.419541] Lustre: Skipped 1 previous similar message [ 1275.360833] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.579501] Lustre: server umount lustre-MDT0001 complete [ 1291.320222] Lustre: server umount lustre-OST0000 complete [ 1311.121439] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1314.393632] Key type lgssc unregistered [ 1314.802052] LNet: 43884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.806714] LNetError: 43884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.819743] LNet: Removed LNI 192.168.201.112@tcp [ 1315.763669] Key type .llcrypt unregistered [ 1315.774265] Key type ._llcrypt unregistered [ 1345.632841] Key type ._llcrypt registered [ 1345.636539] Key type .llcrypt registered [ 1345.784573] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1347.949099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.002324] alg: No test for adler32 (adler32-zlib) [ 1349.335659] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1349.650924] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1351.375847] Key type lgssc registered [ 1353.303749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.040728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.089914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.741717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.483227] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1374.183192] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.208109] LustreError: 45303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1379.306497] LustreError: 45304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.840480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.108240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.102866] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1393.482803] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1400.531612] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1409.335418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.821479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.324314] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1425.386557] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1434.431696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1442.393255] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1444.841630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.845065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.857209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.878871] Lustre: Skipped 1 previous similar message [ 1447.485398] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1448.376151] Lustre: server umount lustre-OST0000 complete [ 1467.375027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1467.386134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.399634] Lustre: Skipped 1 previous similar message [ 1467.406716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.437717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.861778] Lustre: server umount lustre-MDT0000 complete [ 1475.558497] LustreError: 45300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.585538] LustreError: 45300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1476.488219] LustreError: 45285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443296 with bad export cookie 10527952639650935645 [ 1476.494977] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.508033] LustreError: 45285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1483.256883] Lustre: server umount lustre-MDT0001 complete [ 1495.962453] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1498.473736] Key type lgssc unregistered [ 1498.747749] LNet: 49040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.761786] LNetError: 49040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.791895] LNet: Removed LNI 192.168.201.112@tcp [ 1499.621149] Key type .llcrypt unregistered [ 1499.623114] Key type ._llcrypt unregistered [ 1524.583363] Key type ._llcrypt registered [ 1524.587925] Key type .llcrypt registered [ 1524.724677] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1525.806679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.003737] alg: No test for adler32 (adler32-zlib) [ 1527.144704] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1527.397256] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1529.087236] Key type lgssc registered [ 1530.259424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.076608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.088839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.603849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.708528] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1550.818385] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.826838] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1552.468623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.829283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.421370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1566.017167] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1571.374147] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1579.374779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.959647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.100149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1586.777609] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1595.019747] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1603.536701] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1609.004732] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1622.050619] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h50m39s [ 1623.741858] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1628.128811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.134565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.144086] Lustre: Skipped 2 previous similar messages [ 1628.162773] Lustre: Skipped 2 previous similar messages [ 1629.219687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.240100] Lustre: Skipped 2 previous similar messages [ 1630.331132] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1630.343283] Lustre: Skipped 1 previous similar message [ 1632.682040] Lustre: server umount lustre-MDT0000 complete [ 1635.303549] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.324169] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1636.433354] LustreError: 50451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.754827] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1641.448533] LustreError: 51184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.472903] LustreError: 51184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1644.090265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.392662] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1644.761281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.836482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1645.809886] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1649.763945] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1653.875935] Lustre: DEBUG MARKER: Wait for df (36798) ... [ 1655.634600] Lustre: DEBUG MARKER: done [ 1660.393913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1660.410976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.436052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1661.417407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.506837] Lustre: server umount lustre-OST0000 complete [ 1671.648479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.653446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.677856] Lustre: Skipped 1 previous similar message [ 1671.700822] Lustre: Skipped 3 previous similar messages [ 1676.954291] Lustre: server umount lustre-MDT0000 complete [ 1678.835239] LustreError: 51184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.865992] LustreError: 51184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1682.809071] LustreError: 53084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443502 with bad export cookie 15041136374621965238 [ 1682.818385] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.818964] LustreError: 53084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1683.202404] Lustre: server umount lustre-MDT0001 complete [ 1696.797332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1700.054186] Key type lgssc unregistered [ 1700.490021] LNet: 55385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.507510] LNetError: 55385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.550430] LNet: Removed LNI 192.168.201.112@tcp [ 1702.794970] Key type .llcrypt unregistered [ 1702.797925] Key type ._llcrypt unregistered [ 1749.173543] Key type ._llcrypt registered [ 1749.175604] Key type .llcrypt registered [ 1749.286444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1750.450845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1750.501869] alg: No test for adler32 (adler32-zlib) [ 1751.616106] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1751.835855] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1753.527159] Key type lgssc registered [ 1754.700608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1764.577142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1764.588514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.011250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1766.066605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1766.335965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.689072] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1782.768992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.813674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1782.828493] Lustre: 57593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1782.996322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1788.907723] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1797.135629] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1803.943676] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1812.557510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.824275] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1813.194311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.867386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1820.572356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1829.676830] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1835.471657] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h54m14s [ 1837.559552] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1845.648304] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1848.246537] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1848.309358] LustreError: 57601:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1868.412236] Lustre: lustre-MDT0000: Client 0335bcac-bd84-4db1-9496-4a0b1bcb548b (at 192.168.201.12@tcp) reconnecting [ 1888.327135] LustreError: 57601:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1891.355812] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1894.468617] Lustre: DEBUG MARKER: done [ 1900.526429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.547818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.594076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.026507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1906.146506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.150190] Lustre: Skipped 3 previous similar messages [ 1911.273133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.286695] Lustre: Skipped 1 previous similar message [ 1913.311173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1913.635962] Lustre: server umount lustre-OST0000 complete [ 1921.505549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.505846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1921.517891] Lustre: Skipped 1 previous similar message [ 1921.519298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.618540] Lustre: server umount lustre-MDT0000 complete [ 1928.062265] LustreError: 58336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443748 with bad export cookie 4144649445522736868 [ 1928.068823] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1928.072625] LustreError: 58336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1928.349519] Lustre: server umount lustre-MDT0001 complete [ 1940.092042] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1943.205859] Key type lgssc unregistered [ 1943.539772] LNet: 61927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.548875] LNetError: 61927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.597416] LNet: Removed LNI 192.168.201.112@tcp [ 1944.865477] Key type .llcrypt unregistered [ 1944.867062] Key type ._llcrypt unregistered [ 1992.087213] Key type ._llcrypt registered [ 1992.091743] Key type .llcrypt registered [ 1992.236620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1993.309815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.478438] alg: No test for adler32 (adler32-zlib) [ 1994.487896] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1994.724382] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1996.379270] Key type lgssc registered [ 1997.608704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.808210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.860425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.275819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2002.311382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2002.547267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.692518] Lustre: Failing over lustre-MDT0000 [ 2003.938368] Lustre: server umount lustre-MDT0000 complete [ 2010.892408] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:57:09 (1788443829) [ 2017.566877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.854436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.632367] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2030.729361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.852626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2030.888596] Lustre: 64731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.154561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.152650] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2042.858745] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2048.502789] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.138940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.362482] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2056.579232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.704332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2064.034467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2070.594516] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.573594] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2082.272640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2082.284741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.298165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.838150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.849255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.862614] Lustre: Skipped 1 previous similar message [ 2088.343579] Lustre: server umount lustre-MDT0000 complete [ 2091.812478] LustreError: 64721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443911 with bad export cookie 15255695451464221621 [ 2091.821837] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.053938] Lustre: server umount lustre-MDT0001 complete [ 2095.396472] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2099.659826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2107.925659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.455467] Lustre: 63653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443915/real 1788443915] req@00000000c8a97be5 x1875319260749184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.485644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.479151] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443916/real 1788443916] req@000000009498c39f x1875319260749312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.645674] LustreError: 63650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000025a28343 x1875319260750336/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.648950] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443920/real 1788443920] req@000000002edb70a5 x1875319260749440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.127637] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.158462] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2117.306966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.464162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2120.297074] LustreError: 66579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.331720] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.348084] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2122.015640] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2122.338089] Lustre: 63654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443926/real 1788443926] req@000000000dd3f3b6 x1875319260749824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.362113] Lustre: 63654:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2126.819036] LustreError: 68500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.845410] LustreError: 68500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2127.843154] Lustre: 63651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443931/real 1788443931] req@000000007d0cb89c x1875319260749952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.109050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.765173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.931883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2137.065652] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2140.218466] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2149.287182] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.631320] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.657344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2162.669489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.685726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.688215] Lustre: Skipped 2 previous similar messages [ 2165.728494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.749882] Lustre: Skipped 2 previous similar messages [ 2168.963508] Lustre: server umount lustre-MDT0000 complete [ 2170.851438] LustreError: 66579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.861090] LustreError: 66579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2174.051636] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443994 with bad export cookie 15255695451464223896 [ 2174.066903] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.080276] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2174.563612] Lustre: server umount lustre-MDT0001 complete [ 2183.140884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.351257] Lustre: 63652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443996/real 1788443996] req@00000000bcb90982 x1875319260779520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788444012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2192.369677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.393790] Lustre: Skipped 2 previous similar messages [ 2193.735695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.584392] LustreError: 63650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8f79033 x1875319260780544/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.881597] LustreError: 66578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.967575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.015599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2203.435885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2212.292761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.702040] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2217.977314] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2221.026485] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2224.538437] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2231.456105] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2241.507507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.532434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.554122] Lustre: Skipped 2 previous similar messages [ 2241.596886] Lustre: Skipped 2 previous similar messages [ 2245.967924] Lustre: server umount lustre-MDT0000 complete [ 2246.625387] LustreError: 71185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.653534] LustreError: 71185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2250.114720] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444070 with bad export cookie 15255695451464225275 [ 2250.119840] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.124448] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2250.801420] Lustre: server umount lustre-MDT0001 complete [ 2256.897842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2260.140165] Lustre: DEBUG MARKER: files compared the same [ 2272.608873] Lustre: server umount lustre-OST0000 complete [ 2292.452406] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2295.728744] Key type lgssc unregistered [ 2296.049602] LNet: 74476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.056485] LNetError: 74476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.097705] LNet: Removed LNI 192.168.201.112@tcp [ 2297.027568] Key type .llcrypt unregistered [ 2297.029447] Key type ._llcrypt unregistered [ 2324.879612] Key type ._llcrypt registered [ 2324.883177] Key type .llcrypt registered [ 2324.977983] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2326.548640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.557045] alg: No test for adler32 (adler32-zlib) [ 2327.798412] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2328.093858] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2329.911167] Key type lgssc registered [ 2331.556383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.528337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2339.545607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.071550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.484958] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2351.585538] LustreError: 75892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.612487] LustreError: 75892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2354.754455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.024116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2360.859420] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2369.512307] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2376.564341] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2383.220722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.579582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.708431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2389.192738] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2396.634389] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.465215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.476953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.483572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.590202] LustreError: 77715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.766553] Lustre: server umount lustre-OST0000 complete [ 2421.730918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.745438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.760030] Lustre: Skipped 1 previous similar message [ 2421.780913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.787419] Lustre: Skipped 1 previous similar message [ 2425.825210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.825439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.870052] Lustre: Skipped 1 previous similar message [ 2427.262701] Lustre: server umount lustre-MDT0000 complete [ 2430.947915] LustreError: 75892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.977867] LustreError: 75892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2431.589164] LustreError: 75874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444251 with bad export cookie 7865156971654067416 [ 2431.595773] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.609395] LustreError: 75874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2431.998238] Lustre: server umount lustre-MDT0001 complete [ 2444.233505] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2446.450936] Key type lgssc unregistered [ 2446.831890] LNet: 79444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.838104] LNetError: 79444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.885666] LNet: Removed LNI 192.168.201.112@tcp [ 2447.924709] Key type .llcrypt unregistered [ 2447.926441] Key type ._llcrypt unregistered [ 2469.117095] Key type ._llcrypt registered [ 2469.119954] Key type .llcrypt registered [ 2469.218626] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2470.548085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2470.604888] alg: No test for adler32 (adler32-zlib) [ 2471.650865] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2471.904795] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2473.672873] Key type lgssc registered [ 2475.361756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2483.226120] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2483.270386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.918620] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2504.095481] LustreError: 80816:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 2530.719565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.215236] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2545.015782] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2553.548442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.635837] Lustre: *** cfs_fail_loc=706, val=0*** [ 2554.043086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.130219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2559.013156] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2559.459482] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.476959] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2564.580239] LustreError: 80841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.614402] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.655282] LustreError: 80842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2567.084853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.340207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.261668] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2577.855576] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2584.377828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2588.137591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.146174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.148417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.255030] LustreError: 82996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.263029] LustreError: 82996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2593.332774] Lustre: server umount lustre-OST0000 complete [ 2598.369218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2598.373692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.376390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.379292] Lustre: Skipped 2 previous similar messages [ 2598.385616] Lustre: Skipped 2 previous similar messages [ 2603.495234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.499195] Lustre: Skipped 1 previous similar message [ 2603.854375] Lustre: server umount lustre-MDT0000 complete [ 2608.196402] LustreError: 80823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444428 with bad export cookie 6425046924163977421 [ 2608.204838] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.208023] LustreError: 80823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2608.814783] Lustre: server umount lustre-MDT0001 complete [ 2621.025582] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2624.604976] Key type lgssc unregistered [ 2624.888856] LNet: 84437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.894658] LNetError: 84437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.922262] LNet: Removed LNI 192.168.201.112@tcp [ 2625.625477] Key type .llcrypt unregistered [ 2625.627920] Key type ._llcrypt unregistered [ 2650.867930] Key type ._llcrypt registered [ 2650.869774] Key type .llcrypt registered [ 2650.977641] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2652.103145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2652.258689] alg: No test for adler32 (adler32-zlib) [ 2653.522100] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2653.934115] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2655.687221] Key type lgssc registered [ 2656.879112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.279902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.294900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.943597] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2679.652779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.230084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.416318] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.437721] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2685.662389] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2690.542870] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.562421] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2694.758907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.153544] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.186452] LustreError: 86542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2695.198193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.257253] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.265634] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2705.381227] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.410304] LustreError: 86546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2711.012515] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.016916] Lustre: MGS: Client e062fba3-ae01-4917-9239-76ae8dd000a9 (at 0@lo) reconnecting [ 2711.024824] LustreError: 86556:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2711.173369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.262126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2716.633269] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2733.028382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.037653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.046750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.138536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.141078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.144302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.161573] Lustre: Skipped 1 previous similar message [ 2738.823104] Lustre: server umount lustre-OST0000 complete [ 2742.239934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2742.249781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.250455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.362700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.369872] Lustre: Skipped 2 previous similar messages [ 2748.410740] Lustre: server umount lustre-MDT0000 complete [ 2752.118356] Lustre: server umount lustre-MDT0001 complete [ 2755.503052] Lustre: server umount MGS complete [ 2766.552371] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:09:45 (1788444585) [ 2799.029121] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2810.504484] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2845.655517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.858616] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2862.423185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.811569] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2862.897565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2863.013379] Lustre: lustre-MDT0001: new disk, initializing [ 2863.152390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.210259] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.228262] Lustre: 93734:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2863.241640] LustreError: 93734:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2868.275795] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2879.879952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.079565] Lustre: lustre-OST0000: new disk, initializing [ 2880.083489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2880.087456] Lustre: 94501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2880.163620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.028803] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2890.918185] Lustre: 94960:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000074411e39 x1875319951762304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788444726 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2898.939348] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.187922] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.205491] Lustre: MGS: Client b1ec016f-7004-406e-828d-717697badcb2 (at 0@lo) reconnecting [ 2910.259038] Lustre: 93000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.291749] LustreError: 93720:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2910.335792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2910.412901] Lustre: lustre-MDT0000: new disk, initializing [ 2910.560736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.582593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2910.598064] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2911.649494] Lustre: 95571:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000df656685 x1875319951769344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788444747 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2916.304702] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2916.770711] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2920.815502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2920.847917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2920.867128] Lustre: Skipped 1 previous similar message [ 2920.898163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2944.484982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.496243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.518103] Lustre: Skipped 2 previous similar messages [ 2944.541354] Lustre: Skipped 1 previous similar message [ 2954.725388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.737085] Lustre: Skipped 3 previous similar messages [ 2956.255158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2956.452851] Lustre: server umount lustre-OST0000 complete [ 2964.972572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.975802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.009860] Lustre: Skipped 1 previous similar message [ 2975.200410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.201555] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2975.215398] Lustre: Skipped 5 previous similar messages [ 2975.385582] Lustre: server umount lustre-MDT0000 complete [ 2986.887108] Lustre: server umount lustre-MDT0001 complete [ 2990.738851] Lustre: server umount MGS complete [ 3003.704164] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:13:42 (1788444822) [ 3023.389885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3026.145441] Key type lgssc unregistered [ 3026.480848] LNet: 98076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3026.492635] LNetError: 98076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3026.517321] LNet: Removed LNI 192.168.201.112@tcp [ 3027.340665] Key type .llcrypt unregistered [ 3027.347425] Key type ._llcrypt unregistered [ 3046.427496] Key type ._llcrypt registered [ 3046.429650] Key type .llcrypt registered [ 3046.511279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3048.407546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3048.462496] alg: No test for adler32 (adler32-zlib) [ 3049.496426] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3049.613406] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3051.279189] Key type lgssc registered [ 3053.160446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3063.634971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3063.651971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.103281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.865859] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3075.563756] LustreError: 99298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.580637] LustreError: 99298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3077.994904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.248751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3083.771087] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3095.755488] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.822182] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3109.364391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.920128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.979175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3117.211613] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3124.915550] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3136.490872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3136.502851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.517751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3138.532557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3138.533360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.552576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.981069] Lustre: server umount lustre-OST0000 complete [ 3148.770971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.788252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.791781] Lustre: Skipped 1 previous similar message [ 3152.026401] Lustre: server umount lustre-MDT0000 complete [ 3153.890059] LustreError: 100588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.929780] LustreError: 100588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3156.354417] LustreError: 99279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444976 with bad export cookie 15264522138330670432 [ 3156.356522] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.368627] LustreError: 99279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3156.896404] Lustre: server umount lustre-MDT0001 complete [ 3169.968605] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3173.186084] Key type lgssc unregistered [ 3173.648237] LNet: 102835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3173.672114] LNetError: 102835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3174.701205] LNet: Removed LNI 192.168.201.112@tcp [ 3175.880266] Key type .llcrypt unregistered [ 3175.891135] Key type ._llcrypt unregistered [ 3195.756509] Key type ._llcrypt registered [ 3195.758958] Key type .llcrypt registered [ 3195.900348] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3197.249809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3197.276140] alg: No test for adler32 (adler32-zlib) [ 3198.644565] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3198.979539] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3200.703185] Key type lgssc registered [ 3202.007142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.465750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3209.480297] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3209.516753] LustreError: 103922:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3209.528303] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 3209.541960] LustreError: 103918:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3209.551873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.560560] LustreError: 103918:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3211.142135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.170084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.648991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.641646] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3232.156565] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing lsmod [ 3236.056293] LustreError: 105117:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3236.130133] LustreError: 105127:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3236.140799] LustreError: 105117:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4928 [ 3236.145886] LustreError: 105117:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3236.152125] LustreError: 105117:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3236.157582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.666663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.905682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3239.911354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.924050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.953968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.954880] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.959186] Lustre: Skipped 1 previous similar message [ 3241.965835] Lustre: Skipped 1 previous similar message [ 3244.000995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.018809] Lustre: Skipped 2 previous similar messages [ 3247.010485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.026284] Lustre: Skipped 2 previous similar messages [ 3252.195373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.203637] Lustre: Skipped 2 previous similar messages [ 3253.215422] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3253.441255] Lustre: server umount lustre-MDT0000 complete [ 3257.099452] LustreError: 103935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445077 with bad export cookie 15372497850986702363 [ 3257.102274] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.110080] LustreError: 103935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3257.312514] LustreError: 105315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.315246] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.327154] LustreError: 105315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3258.529058] LustreError: 104390:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e02d5d2e x1875320524265728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3258.722351] Lustre: server umount lustre-MDT0001 complete [ 3265.095971] Lustre: server umount lustre-OST0000 complete [ 3275.564460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.232332] LustreError: 106194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.399225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.391680] LustreError: 106195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.872285] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3286.503236] LustreError: 106194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.655905] LustreError: 106194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.683050] LustreError: 106194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3293.560790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.179150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.708925] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3308.626625] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3315.694627] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3322.792413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.934897] LustreError: 108015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.244845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.518136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3331.375798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3339.393830] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3348.962406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.967344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.981503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.989172] Lustre: Skipped 1 previous similar message [ 3349.989216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.223145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3360.511861] Lustre: server umount lustre-OST0000 complete [ 3365.348055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.352667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.361434] Lustre: Skipped 2 previous similar messages [ 3365.364674] Lustre: Skipped 6 previous similar messages [ 3370.958527] Lustre: server umount lustre-MDT0000 complete [ 3374.913636] LustreError: 108690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445194 with bad export cookie 15372497850986703049 [ 3374.916040] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.921822] LustreError: 108690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3375.530367] Lustre: server umount lustre-MDT0001 complete [ 3388.283723] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3392.066728] Key type lgssc unregistered [ 3392.548317] LNet: 109734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3392.556787] LNetError: 109734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3392.571969] LNet: Removed LNI 192.168.201.112@tcp [ 3393.779502] Key type .llcrypt unregistered [ 3393.783300] Key type ._llcrypt unregistered [ 3417.543708] Key type ._llcrypt registered [ 3417.548709] Key type .llcrypt registered [ 3417.727927] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3419.542571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3419.682916] alg: No test for adler32 (adler32-zlib) [ 3421.307830] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3421.767023] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3423.663217] Key type lgssc registered [ 3425.645494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3434.314633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3434.329407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.811417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.325261] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3445.222954] LustreError: 111177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.243578] LustreError: 111177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3450.338351] LustreError: 111178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.080789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.696368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.137569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3467.447227] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3474.871278] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3483.521133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.313476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.274858] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3500.438485] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.906393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3510.313466] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3514.531884] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3521.506344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.514944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3521.522202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.420381] Lustre: server umount lustre-OST0000 complete [ 3529.696189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3529.704754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.720092] Lustre: Skipped 1 previous similar message [ 3529.728769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.737376] Lustre: Skipped 1 previous similar message [ 3531.746276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.760753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.264363] Lustre: server umount lustre-MDT0000 complete [ 3536.874362] LustreError: 111178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.897913] LustreError: 111178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3539.017155] LustreError: 111913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445359 with bad export cookie 16216239514659076572 [ 3539.028215] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.034891] LustreError: 111913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3540.140069] Lustre: server umount lustre-MDT0001 complete [ 3551.925516] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3554.979238] Key type lgssc unregistered [ 3555.337471] LNet: 114822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3555.348603] LNetError: 114822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3555.379122] LNet: Removed LNI 192.168.201.112@tcp [ 3556.468823] Key type .llcrypt unregistered [ 3556.471218] Key type ._llcrypt unregistered [ 3576.680120] Key type ._llcrypt registered [ 3576.681864] Key type .llcrypt registered [ 3576.818530] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3578.456903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3578.826857] alg: No test for adler32 (adler32-zlib) [ 3580.311728] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3580.737895] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3582.479949] Key type lgssc registered [ 3584.034032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.151960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.189953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.960888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.966907] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3604.453184] LustreError: 116045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.474850] LustreError: 116045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3608.149553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.406357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3613.478928] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3621.008618] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3627.076518] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3633.783217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.134723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3639.312124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3640.600450] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3650.588649] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3662.824503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3662.832206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.841168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.845718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.849530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.851698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3666.932340] Lustre: server umount lustre-OST0000 complete [ 3674.089877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.098361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.114070] Lustre: Skipped 1 previous similar message [ 3674.118490] Lustre: Skipped 1 previous similar message [ 3676.895248] Lustre: server umount lustre-MDT0000 complete [ 3679.199822] LustreError: 116044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.205302] LustreError: 116044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3680.903843] LustreError: 116778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445500 with bad export cookie 16940418821529468609 [ 3680.912293] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.919866] LustreError: 116778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3681.393620] Lustre: server umount lustre-MDT0001 complete [ 3692.802798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3695.419539] Key type lgssc unregistered [ 3695.702636] LNet: 119582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3695.714590] LNetError: 119582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3695.726343] LNet: Removed LNI 192.168.201.112@tcp [ 3696.356925] Key type .llcrypt unregistered [ 3696.361979] Key type ._llcrypt unregistered [ 3718.769276] Key type ._llcrypt registered [ 3718.771593] Key type .llcrypt registered [ 3718.925131] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3720.259812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.299172] alg: No test for adler32 (adler32-zlib) [ 3721.390075] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3721.628514] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3723.279684] Key type lgssc registered [ 3724.720809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3732.337982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3732.350919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.698454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.660954] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3743.225728] LustreError: 121002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.246874] LustreError: 121002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3744.301942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.587423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.792517] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3759.723395] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3767.542992] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3777.791398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.483268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.533336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3787.043502] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3794.739731] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3802.388228] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3806.431722] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3806.437238] Lustre: Skipped 2 previous similar messages [ 3810.173283] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3810.181416] Lustre: Skipped 2 previous similar messages [ 3810.438615] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3810.465608] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3815.634641] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3815.642670] Lustre: Skipped 2 previous similar messages [ 3817.412961] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp [ 3821.517277] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.12/24@tcp in log lustre-MDT0000 [ 3821.525305] Lustre: Skipped 2 previous similar messages [ 3824.749504] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 3824.763654] Lustre: Skipped 1 previous similar message [ 3831.786301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3831.789144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.791061] LustreError: Skipped 1 previous similar message [ 3831.802064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.015551] Lustre: server umount lustre-OST0000 complete [ 3836.897345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.902381] LustreError: Skipped 4 previous similar messages [ 3836.907516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.924852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.992618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.994446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.003745] Lustre: server umount lustre-MDT0000 complete [ 3846.121319] LustreError: 121003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.874337] LustreError: 121735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445666 with bad export cookie 4573403456869900680 [ 3846.879073] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.888579] LustreError: 121735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3856.537797] Lustre: server umount lustre-MDT0001 complete [ 3867.949939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3870.521565] Key type lgssc unregistered [ 3870.838740] LNet: 125699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3870.851545] LNetError: 125699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3870.877103] LNet: Removed LNI 192.168.201.112@tcp [ 3871.818847] Key type .llcrypt unregistered [ 3871.822128] Key type ._llcrypt unregistered [ 3900.930847] Key type ._llcrypt registered [ 3900.939346] Key type .llcrypt registered [ 3901.170371] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3903.280829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3903.360235] alg: No test for adler32 (adler32-zlib) [ 3904.471748] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3904.682523] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3906.352558] Key type lgssc registered [ 3907.905135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3916.881407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3916.917288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.330832] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3918.350544] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.12/24@tcp [ 3918.376867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.995181] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3928.545634] LustreError: 127260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.583824] LustreError: 127260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3932.883491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.223560] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3933.257195] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 3933.326396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3937.591463] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3945.132237] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.409788] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3958.520696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.998317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.092718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3965.780840] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3973.937743] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.944938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3987.947677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.955722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.963352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.966880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.970047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.110471] Lustre: server umount lustre-OST0000 complete [ 3999.201568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.202985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.219151] Lustre: Skipped 1 previous similar message [ 3999.240388] Lustre: Skipped 2 previous similar messages [ 4004.208333] Lustre: server umount lustre-MDT0000 complete [ 4004.326297] LustreError: 127260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.349609] LustreError: 127260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4007.968167] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445828 with bad export cookie 16441599031519714843 [ 4007.975157] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.980564] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4008.402259] Lustre: server umount lustre-MDT0001 complete [ 4019.703984] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4023.383544] Key type lgssc unregistered [ 4023.738841] LNet: 130854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4023.744328] LNetError: 130854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4023.765350] LNet: Removed LNI 192.168.201.112@tcp [ 4024.758651] Key type .llcrypt unregistered [ 4024.760215] Key type ._llcrypt unregistered [ 4054.460942] Key type ._llcrypt registered [ 4054.468174] Key type .llcrypt registered [ 4054.621538] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4070.824880] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4071.786936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4071.984194] alg: No test for adler32 (adler32-zlib) [ 4073.268318] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4073.708130] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4075.463181] Key type lgssc registered [ 4076.975710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.105475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.144268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.605578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4116.639305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4116.786469] Lustre: lustre-MDT0000: new disk, initializing [ 4116.898567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.920946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4122.207067] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4133.445375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.564796] Lustre: 134259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4133.650788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4133.653789] Lustre: Skipped 1 previous similar message [ 4133.743243] Lustre: lustre-MDT0001: new disk, initializing [ 4133.865823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.927070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4133.943604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.020961] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4150.018320] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4156.334575] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4163.747325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4163.914557] Lustre: lustre-OST0000: new disk, initializing [ 4163.916662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4163.919216] Lustre: 136226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4163.967630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.391989] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4173.407487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4173.418728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4173.590685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4183.573968] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4190.353735] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4190.696984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4197.011354] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4197.279168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4204.512148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4204.516766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.538905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.585318] Lustre: Skipped 1 previous similar message [ 4206.043429] Lustre: server umount lustre-OST0000 complete [ 4210.659451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4210.672251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.685266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.693630] Lustre: Skipped 1 previous similar message [ 4214.754355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.780046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.516588] Lustre: server umount lustre-MDT0000 complete [ 4220.769888] LustreError: 134251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446040 with bad export cookie 5516920500194917744 [ 4220.780636] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.788222] LustreError: 134251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4221.214284] Lustre: server umount lustre-MDT0001 complete [ 4229.358181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4229.600393] LustreError: 138445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.613171] LustreError: 138445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4229.658763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.721936] LustreError: 138446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.107544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4239.841966] LustreError: 138445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.962038] LustreError: 138446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.345422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.181494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.272264] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4261.372776] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4268.983611] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4276.477170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4276.821561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.679733] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4291.551037] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4296.072405] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4300.658379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4305.856617] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4306.159985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4316.069844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4316.309557] Lustre: lustre-OST0001: new disk, initializing [ 4316.313067] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4316.319905] Lustre: 141845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4316.446415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4321.908959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4321.917026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4322.082740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4322.547867] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4331.015432] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4331.283175] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4336.249995] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4336.454548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4342.490112] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9365d82ff000.ost_server_uuid 50 [ 4344.482541] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9365d82ff000.ost_server_uuid in IDLE state after 0 sec [ 4352.992326] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4352.997246] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.005177] LustreError: Skipped 1 previous similar message [ 4353.008148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4353.018375] Lustre: Skipped 1 previous similar message [ 4353.029631] Lustre: Skipped 2 previous similar messages [ 4358.975471] Lustre: server umount lustre-OST0001 complete [ 4361.211135] LustreError: 140267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.230420] LustreError: 140267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4363.239437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.257227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.287574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.295766] Lustre: Skipped 2 previous similar messages [ 4371.428547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4371.429493] LustreError: 142142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.446320] Lustre: Skipped 3 previous similar messages [ 4371.472492] LustreError: 142142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4377.057219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4377.239791] Lustre: server umount lustre-OST0000 complete [ 4384.226077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4384.249105] LustreError: Skipped 1 previous similar message [ 4384.251113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.294061] Lustre: Skipped 1 previous similar message [ 4389.149917] Lustre: server umount lustre-MDT0000 complete [ 4391.905202] LustreError: 138446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.953048] LustreError: 138446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4394.052485] LustreError: 139180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446214 with bad export cookie 5516920500194918626 [ 4394.056317] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.058327] LustreError: 139180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4394.551579] Lustre: server umount lustre-MDT0001 complete [ 4416.697319] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4419.790314] Key type lgssc unregistered [ 4420.096661] LNet: 144853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4420.107462] LNetError: 144853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4420.125819] LNet: Removed LNI 192.168.201.112@tcp [ 4420.826309] Key type .llcrypt unregistered [ 4420.828786] Key type ._llcrypt unregistered [ 4460.792577] Key type ._llcrypt registered [ 4460.795290] Key type .llcrypt registered [ 4460.871330] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4476.575761] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4478.175457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4478.510310] alg: No test for adler32 (adler32-zlib) [ 4479.599565] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4479.820785] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4481.463173] Key type lgssc registered [ 4482.661262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4516.930275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4516.953317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.194780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4518.219272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4518.288667] Lustre: lustre-MDT0000: new disk, initializing [ 4518.392555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.415914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4523.437387] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4534.995795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.169677] Lustre: 149007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4535.313562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4535.323332] Lustre: Skipped 1 previous similar message [ 4535.453505] Lustre: lustre-MDT0001: new disk, initializing [ 4535.563581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4535.629274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4535.647352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4541.625947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4551.835748] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4557.649869] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4564.323704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.722970] Lustre: lustre-OST0000: new disk, initializing [ 4564.728984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4564.745445] Lustre: 150976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4564.840302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.173627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4570.184624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4570.293891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4570.807708] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4579.354459] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4583.221013] Lustre: Failing over lustre-OST0000 [ 4583.424956] Lustre: server umount lustre-OST0000 complete [ 4602.336755] Lustre: 147232:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446406/real 1788446406] req@000000006f7f4ece x1875321866453760/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788446422 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4602.339461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.365829] Lustre: 147232:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4602.396841] Lustre: Skipped 1 previous similar message [ 4603.023945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4603.570173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.583613] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4607.497103] Lustre: 147232:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446411/real 1788446411] req@00000000c45b45a2 x1875321866454912/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788446427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4607.554050] Lustre: 147232:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4610.835523] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4612.063864] Lustre: 147234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446416/real 1788446416] req@0000000049ef992e x1875321866456064/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788446432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4612.112748] Lustre: 147234:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4612.675842] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4612.757797] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4612.772036] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4614.849618] Lustre: Failing over lustre-MDT0000 [ 4615.247728] Lustre: server umount lustre-MDT0000 complete [ 4617.698828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4617.711444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.871200] Lustre: 147234:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446421/real 1788446421] req@00000000b594b6ac x1875321866457088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788446437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4617.899678] Lustre: 147234:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4622.925450] LustreError: 149013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.946427] LustreError: 149013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4628.039697] LustreError: 149013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.052017] LustreError: 149013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4633.160031] LustreError: 149012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.203865] LustreError: 149012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4634.591182] Lustre: 147232:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446438/real 1788446438] req@000000002d4ae2ca x1875321866466816/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788446454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4634.627242] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.302891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.264848] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.282616] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4644.417474] LustreError: 150169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.432960] LustreError: 150169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4645.118703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.162243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4649.533218] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4649.747618] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4650.473521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4650.481988] Lustre: Skipped 1 previous similar message [ 4650.500016] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4650.546938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4659.171135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4659.178175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.198491] Lustre: Skipped 2 previous similar messages [ 4659.213600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.710045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4660.719342] Lustre: Skipped 1 previous similar message [ 4663.812022] Lustre: server umount lustre-OST0000 complete [ 4670.948467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4670.948513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.956794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.988535] Lustre: Skipped 2 previous similar messages [ 4674.456530] Lustre: server umount lustre-MDT0000 complete [ 4676.069285] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.092383] LustreError: 149014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4678.460865] LustreError: 149810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446498 with bad export cookie 10057444550830672882 [ 4678.470543] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.484876] LustreError: 149810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4679.036109] Lustre: server umount lustre-MDT0001 complete [ 4690.833708] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4694.026284] Key type lgssc unregistered [ 4694.318472] LNet: 154635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4694.328598] LNetError: 154635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4694.346485] LNet: Removed LNI 192.168.201.112@tcp [ 4695.360238] Key type .llcrypt unregistered [ 4695.365086] Key type ._llcrypt unregistered [ 4733.313440] Key type ._llcrypt registered [ 4733.323311] Key type .llcrypt registered [ 4733.414968] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4734.584778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4734.699277] alg: No test for adler32 (adler32-zlib) [ 4735.853557] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4736.133132] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4737.863243] Key type lgssc registered [ 4739.410829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4747.493775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.525576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.065481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.883189] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4759.541256] LustreError: 156156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4759.559478] LustreError: 156156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4763.785763] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4764.651443] LustreError: 156157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.317886] LustreError: 156185:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4768.343351] LustreError: 156185:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4768.355861] LustreError: 156185:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4768.715215] Lustre: server umount lustre-MDT0000 complete [ 4784.401892] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4794.405887] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4829.931050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.121683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4830.147880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4830.218664] Lustre: lustre-MDT0000: new disk, initializing [ 4830.309030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.318446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4834.333457] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4845.847392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.055870] Lustre: 159499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4846.179911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4846.189071] Lustre: Skipped 1 previous similar message [ 4846.343869] Lustre: lustre-MDT0001: new disk, initializing [ 4846.526345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4846.618984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4846.633590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4852.258738] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4862.566157] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4866.857865] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4874.288560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.760256] Lustre: lustre-OST0000: new disk, initializing [ 4874.764925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.775892] Lustre: 161468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4874.894043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.332807] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4885.101776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4885.111745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4885.291598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4892.485654] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4896.222978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4897.671656] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:45:16 (1788446716) [ 5117.920073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5117.930042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.944086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5120.998948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.013753] Lustre: Skipped 1 previous similar message [ 5121.021871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.564867] Lustre: server umount lustre-MDT0000 complete [ 5126.115781] LustreError: 160580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.130669] LustreError: 160580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5127.506204] LustreError: 159492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446947 with bad export cookie 12306418813516962078 [ 5127.510506] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.512760] LustreError: 159492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5131.232517] LustreError: 160515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.233082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.255799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5131.258649] Lustre: Skipped 2 previous similar messages [ 5133.823341] Lustre: server umount lustre-MDT0001 complete [ 5152.223151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5152.374317] Lustre: server umount lustre-OST0000 complete [ 5160.619663] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5169.637411] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5203.285741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5203.552912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5203.565438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5203.613600] Lustre: lustre-MDT0000: new disk, initializing [ 5203.664476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.673752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5207.385431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5217.292712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.417041] Lustre: 166126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5217.470840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5217.473949] Lustre: Skipped 1 previous similar message [ 5217.549230] Lustre: lustre-MDT0001: new disk, initializing [ 5217.631992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5217.655217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5217.659276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5221.514986] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5230.500296] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5235.998762] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5242.365547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.636679] Lustre: lustre-OST0000: new disk, initializing [ 5242.639901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5242.645592] Lustre: 168093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5242.773236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.167285] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5249.660502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5249.676629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5249.836794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5257.500603] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5261.611907] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5261.819619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5266.188676] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5266.452448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5270.500570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5270.513746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.524305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5270.527326] Lustre: Skipped 1 previous similar message [ 5270.541606] Lustre: Skipped 2 previous similar messages [ 5275.618385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.627212] Lustre: Skipped 1 previous similar message [ 5283.295310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5283.617798] Lustre: server umount lustre-OST0000 complete [ 5289.440813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5289.448483] LustreError: Skipped 1 previous similar message [ 5289.454197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.468320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.473204] Lustre: Skipped 2 previous similar messages [ 5302.239226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5302.464479] Lustre: server umount lustre-MDT0000 complete [ 5306.304466] LustreError: 166118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447126 with bad export cookie 12306418813517120908 [ 5306.317436] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.326574] LustreError: 166118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5306.828768] Lustre: server umount lustre-MDT0001 complete [ 5314.991467] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:52:13 (1788447133) [ 5332.793597] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5345.012298] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5381.463425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5381.621196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5381.638722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5381.707547] Lustre: lustre-MDT0000: new disk, initializing [ 5381.776536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.790563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5385.832682] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5395.602290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.692440] Lustre: 172854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5395.900598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5400.617468] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5410.362991] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5416.261029] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5422.997119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.183275] Lustre: lustre-OST0000: new disk, initializing [ 5423.185611] Lustre: Skipped 1 previous similar message [ 5423.188521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5423.191889] Lustre: Skipped 2 previous similar messages [ 5423.194935] Lustre: 174830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5429.301405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5429.313533] Lustre: Skipped 1 previous similar message [ 5429.324866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5429.464690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5429.557503] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5437.647131] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.090918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.101308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.106365] LustreError: Skipped 1 previous similar message [ 5445.107903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.107910] Lustre: Skipped 6 previous similar messages [ 5445.133234] Lustre: Skipped 2 previous similar messages [ 5451.284807] Lustre: server umount lustre-OST0000 complete [ 5457.389056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5462.181304] Lustre: server umount lustre-MDT0000 complete [ 5465.581522] LustreError: 175456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.610665] LustreError: 175456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5466.502666] LustreError: 172847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447286 with bad export cookie 12306418813517121790 [ 5466.504376] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5466.518778] LustreError: 172847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5466.940721] Lustre: server umount lustre-MDT0001 complete [ 5474.060175] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:54:53 (1788447293) [ 5489.742435] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5499.192323] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5537.703410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.075043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5538.112500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5538.209545] Lustre: lustre-MDT0000: new disk, initializing [ 5538.297912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.313355] Lustre: Skipped 2 previous similar messages [ 5538.340298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5543.575513] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5553.251371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.424438] Lustre: 179153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5553.692601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5558.443269] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5568.516591] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5574.494725] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.158026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.399135] Lustre: 181129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5587.518593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5587.549198] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5587.708864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5597.208438] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5608.416882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5608.428660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.435648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.452517] Lustre: Skipped 3 previous similar messages [ 5608.468543] Lustre: Skipped 7 previous similar messages [ 5609.651896] Lustre: server umount lustre-OST0000 complete [ 5627.359164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5627.662624] Lustre: server umount lustre-MDT0000 complete [ 5628.900970] LustreError: 179163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.920283] LustreError: 179163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5632.274883] LustreError: 179144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447452 with bad export cookie 12306418813517122875 [ 5632.287248] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.293298] LustreError: 179144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5632.872596] Lustre: server umount lustre-MDT0001 complete [ 5647.516798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5649.482499] Key type lgssc unregistered [ 5650.054775] LNet: 182991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5650.067547] LNetError: 182991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5650.084870] LNet: Removed LNI 192.168.201.112@tcp [ 5651.252280] Key type .llcrypt unregistered [ 5651.255144] Key type ._llcrypt unregistered [ 5677.342335] Key type ._llcrypt registered [ 5677.344266] Key type .llcrypt registered [ 5677.414975] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5678.588775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5678.856326] alg: No test for adler32 (adler32-zlib) [ 5680.062610] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5680.277459] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5681.975203] Key type lgssc registered [ 5683.163025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.542820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5689.573457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.890082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.463649] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5700.073339] LustreError: 184408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.095642] LustreError: 184408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5704.112466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.799414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.509362] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5718.097956] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5723.846326] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5729.500545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.783957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.323208] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5736.997977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5744.587191] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5767.648480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5767.660535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.684283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.701397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.712819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.719281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.755713] Lustre: Skipped 1 previous similar message [ 5771.925475] Lustre: server umount lustre-OST0000 complete [ 5776.352549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5776.357765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.377071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.937698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.938491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.210898] Lustre: server umount lustre-MDT0000 complete [ 5785.057372] LustreError: 185591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.089499] LustreError: 185591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5788.319984] LustreError: 184387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447608 with bad export cookie 16950635797240992373 [ 5788.326199] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.330334] LustreError: 184387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5788.626285] Lustre: server umount lustre-MDT0001 complete [ 5800.138381] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5802.760738] Key type lgssc unregistered [ 5803.014130] LNet: 187966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.020889] LNetError: 187966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5804.070812] LNet: Removed LNI 192.168.201.112@tcp [ 5805.238811] Key type .llcrypt unregistered [ 5805.244646] Key type ._llcrypt unregistered [ 5830.552190] Key type ._llcrypt registered [ 5830.554305] Key type .llcrypt registered [ 5830.681871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5832.304959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5832.534184] alg: No test for adler32 (adler32-zlib) [ 5833.668924] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5833.987480] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5835.695279] Key type lgssc registered [ 5836.521384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.355084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.387476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.013478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.901197] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5855.202184] LustreError: 189396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.228794] LustreError: 189396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5857.786862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.095861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.590775] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5870.065442] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5875.647927] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5882.001350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.315675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5888.149736] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5890.494405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5895.388695] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5903.453185] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 5903.842112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5903.846791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.889565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.900903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.903866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.917867] Lustre: Skipped 2 previous similar messages [ 5907.876324] Lustre: server umount lustre-OST0000 complete [ 5916.404592] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9365c30e1000.ost_server_uuid 50 [ 5918.080092] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5931.489725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.497772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5931.506637] Lustre: Skipped 1 previous similar message [ 5937.479419] Lustre: server umount lustre-MDT0000 complete [ 5937.638691] LustreError: 189396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.652296] LustreError: 189396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5940.967980] LustreError: 189377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447761 with bad export cookie 9790520755659170307 [ 5940.978095] LustreError: 189377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5940.978473] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5941.403076] Lustre: server umount lustre-MDT0001 complete [ 5948.841904] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:02:47 (1788447767) [ 5955.864987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.216692] LustreError: 193311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.290471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.329205] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5961.706628] LustreError: 193312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.824498] LustreError: 193311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.542717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.842197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.498390] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5980.733970] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.706832] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5994.057257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.395811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.580892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6001.777284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6010.037537] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6016.882945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6017.047280] Lustre: lustre-OST0001: new disk, initializing [ 6017.055890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6017.061906] Lustre: 196154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6017.155091] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.326944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6023.338683] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6023.446079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6024.482049] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6033.651902] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.476926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.497400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.546207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.555364] Lustre: Skipped 3 previous similar messages [ 6044.134511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.143415] Lustre: Skipped 4 previous similar messages [ 6050.271305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6050.374120] Lustre: server umount lustre-OST0000 complete [ 6054.370393] LustreError: 195134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6054.399083] LustreError: 195134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6056.549350] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6056.762266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6061.736913] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6061.959749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6074.857230] LustreError: 195495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.876441] LustreError: 195495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6079.976223] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.982473] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.984038] LustreError: Skipped 1 previous similar message [ 6079.989447] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6079.996745] Lustre: Skipped 2 previous similar messages [ 6080.016355] Lustre: Skipped 6 previous similar messages [ 6082.112983] Lustre: server umount lustre-OST0001 complete [ 6090.209247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.224248] Lustre: Skipped 1 previous similar message [ 6092.593245] Lustre: server umount lustre-MDT0000 complete [ 6096.507300] LustreError: 194044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447916 with bad export cookie 9790520755659171385 [ 6096.516721] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.827524] Lustre: server umount lustre-MDT0001 complete [ 6120.342973] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:05:38 (1788447938) [ 6127.074747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.314955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6127.352210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6127.553926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.209751] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6141.670901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.804957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6141.853819] Lustre: 199596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6146.513160] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6153.995604] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6159.483950] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6165.316968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6165.456780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6171.481609] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6172.350350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6176.578915] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6183.403099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6183.496710] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6190.359305] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6198.277628] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6202.344166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.349526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.377504] Lustre: Skipped 1 previous similar message [ 6202.405516] Lustre: Skipped 3 previous similar messages [ 6206.717258] Lustre: server umount lustre-OST0000 complete [ 6207.474221] LustreError: 201447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.496464] LustreError: 201447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6222.832065] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6222.841422] LustreError: Skipped 1 previous similar message [ 6228.288285] Lustre: server umount lustre-OST0001 complete [ 6238.177439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.193220] Lustre: Skipped 6 previous similar messages [ 6238.621174] Lustre: server umount lustre-MDT0000 complete [ 6242.223973] LustreError: 200343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448062 with bad export cookie 9790520755659172792 [ 6242.227241] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.233190] LustreError: 200343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6242.612738] Lustre: server umount lustre-MDT0001 complete [ 6263.429906] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:08:02 (1788448082) [ 6279.742855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6290.054385] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6322.333248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.518284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6322.539642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6322.611388] Lustre: lustre-MDT0000: new disk, initializing [ 6322.678241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.681293] Lustre: Skipped 3 previous similar messages [ 6322.691653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6326.700390] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6336.350232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.518727] Lustre: 207692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6336.663965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6336.678224] Lustre: Skipped 1 previous similar message [ 6336.850311] Lustre: lustre-MDT0001: new disk, initializing [ 6337.066420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6337.078203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6342.088556] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6352.651557] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6357.868963] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.639866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.795486] Lustre: lustre-OST0000: new disk, initializing [ 6362.797308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6362.799702] Lustre: 209658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6369.116637] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6370.374760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6370.387313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6370.559150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6378.068286] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6382.487612] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6382.677619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6387.865279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6388.081127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6391.264534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.269060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.275148] LustreError: Skipped 1 previous similar message [ 6391.281202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6391.301535] Lustre: Skipped 5 previous similar messages [ 6391.324640] Lustre: Skipped 1 previous similar message [ 6396.896642] Lustre: server umount lustre-OST0000 complete [ 6408.375835] Lustre: server umount lustre-MDT0000 complete [ 6411.240688] LustreError: 209657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448231 with bad export cookie 9790520755659174353 [ 6411.244150] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.249328] LustreError: 209657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6411.602254] Lustre: server umount lustre-MDT0001 complete [ 6420.146636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.673747] LustreError: 211877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.686683] LustreError: 211877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6425.691570] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6434.355598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.511478] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6448.700709] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6454.490144] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6460.887258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6461.270789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.277871] Lustre: Skipped 4 previous similar messages [ 6468.385364] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6477.105150] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.798458] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6483.086459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6488.145954] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6488.382154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6491.106963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6491.120412] LustreError: Skipped 1 previous similar message [ 6491.130417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.153072] Lustre: Skipped 2 previous similar messages [ 6496.872751] Lustre: server umount lustre-OST0000 complete [ 6503.510799] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6509.301527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6538.461082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.947815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.964825] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6546.455664] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6557.455462] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.848148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6560.362029] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.378943] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6565.866270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.880225] Lustre: Skipped 8 previous similar messages [ 6570.465490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6570.481815] LustreError: Skipped 1 previous similar message [ 6571.962310] Lustre: server umount lustre-OST0000 complete [ 6581.860843] Lustre: server umount lustre-MDT0000 complete [ 6586.336670] LustreError: 211873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.356958] LustreError: 211873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6586.414553] LustreError: 211858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448406 with bad export cookie 9790520755659175235 [ 6586.418773] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.425693] LustreError: 211858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6586.726779] Lustre: server umount lustre-MDT0001 complete [ 6597.914808] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:13:36 (1788448416) [ 6604.473120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.745349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6618.389168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6624.168184] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6632.078214] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6638.332289] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.603556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.379279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6658.339567] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6663.560486] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6663.745434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6668.533176] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6668.709781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6675.190588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.356952] Lustre: lustre-OST0001: new disk, initializing [ 6675.360890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6675.367107] Lustre: 221215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6681.227622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6681.243612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6681.359445] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6681.763828] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6691.607751] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.604916] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6696.876784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6701.219196] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6701.368380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.168215] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.171516] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.201550] Lustre: Skipped 6 previous similar messages [ 6709.403965] Lustre: server umount lustre-OST0001 complete [ 6715.859213] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6716.085297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6721.115931] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6721.356000] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6749.443919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.761824] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.772368] Lustre: Skipped 5 previous similar messages [ 6754.809182] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6754.830531] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6754.832964] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6754.851537] Lustre: Skipped 1 previous similar message [ 6755.821944] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6762.874467] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.486476] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6772.743024] Lustre: server umount lustre-OST0001 complete [ 6795.405202] Lustre: server umount lustre-OST0000 complete [ 6805.253451] Lustre: server umount lustre-MDT0000 complete [ 6809.263213] LustreError: 221214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448629 with bad export cookie 9790520755659176201 [ 6809.279822] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.299772] LustreError: 221214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6815.858613] Lustre: server umount lustre-MDT0001 complete [ 6840.325974] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:17:39 (1788448659) [ 6847.275393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.586492] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6847.627719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6852.590363] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6862.195061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.288485] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6862.331746] Lustre: 226460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6867.668503] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6875.868465] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6881.284454] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6887.708590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.914790] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6894.338980] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6902.725986] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6911.695163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6911.805717] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6918.969311] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6926.549448] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6927.854418] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6933.193341] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6933.483717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6938.152575] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6938.364789] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6943.626340] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9365c304b800.ost_server_uuid 50 [ 6945.138712] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9365c304b800.ost_server_uuid in IDLE state after 0 sec [ 6946.063376] Lustre: Permanently deactivating lustre-OST0001 [ 6946.091613] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6949.425888] Lustre: Permanently reactivating lustre-OST0001 [ 6956.000250] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6956.007598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6956.008895] LustreError: Skipped 4 previous similar messages [ 6956.022058] Lustre: Skipped 16 previous similar messages [ 6959.918644] Lustre: server umount lustre-OST0001 complete [ 6961.127296] LustreError: 228305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.154526] LustreError: 228305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 6965.221801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.245930] Lustre: Skipped 8 previous similar messages [ 6969.368530] Lustre: server umount lustre-OST0000 complete [ 6988.255141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.553412] Lustre: server umount lustre-MDT0000 complete [ 6992.508627] LustreError: 226452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448812 with bad export cookie 9790520755659177489 [ 6992.512653] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.516519] LustreError: 226452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6992.927633] Lustre: server umount lustre-MDT0001 complete [ 7017.153857] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:20:36 (1788448836) [ 7024.048845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.505289] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7024.537861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7024.545989] Lustre: Skipped 5 previous similar messages [ 7029.350601] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7038.075251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.178056] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7038.215980] Lustre: 233153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7043.447527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7050.144695] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7055.342391] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7062.525686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.703147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7062.748146] Lustre: Permanently deactivating lustre-OST0000 [ 7062.794053] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7069.315523] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7071.780677] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7076.981238] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.802570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7083.940728] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7090.481939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7092.456138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7098.159928] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7099.911323] Lustre: Permanently reactivating lustre-OST0000 [ 7099.937783] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7099.949647] Lustre: Skipped 2 previous similar messages [ 7108.241526] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7108.243252] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 0000000092bc71fb, cur 1788448928 deadline 1788448921 last 1788448891 [ 7108.259416] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7108.295797] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7108.369687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7120.893508] Lustre: server umount lustre-OST0001 complete [ 7130.946212] Lustre: server umount lustre-OST0000 complete [ 7140.794250] Lustre: server umount lustre-MDT0000 complete [ 7145.149417] LustreError: 233146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448965 with bad export cookie 9790520755659179155 [ 7145.151405] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7145.163637] LustreError: 233146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7145.762135] Lustre: server umount lustre-MDT0001 complete [ 7157.448965] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7160.337337] Key type lgssc unregistered [ 7160.672862] LNet: 238042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7160.695815] LNetError: 238042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7160.740349] LNet: Removed LNI 192.168.201.112@tcp [ 7161.866129] Key type .llcrypt unregistered [ 7161.872190] Key type ._llcrypt unregistered [ 7185.176376] Key type ._llcrypt registered [ 7185.178491] Key type .llcrypt registered [ 7185.296972] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7186.420937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7186.432170] alg: No test for adler32 (adler32-zlib) [ 7187.870442] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7188.139989] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7189.839201] Key type lgssc registered [ 7191.096210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.718051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.750489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.437674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.067553] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7214.561387] LustreError: 239508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.568562] LustreError: 239508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7217.252711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.516179] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7217.589885] Lustre: 239496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7217.606484] Lustre: Permanently deactivating lustre-MDT0001 [ 7217.640815] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7218.087520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.326263] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7224.886157] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7232.115412] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.986288] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.648761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.958671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.083965] LustreError: 241352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.094100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7250.467440] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7254.502988] LustreError: 241352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.678881] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.624569] LustreError: 241353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.528080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7263.733507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7268.883957] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7268.908737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7270.192816] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7278.578321] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7282.736694] Lustre: Permanently reactivating lustre-MDT0001 [ 7282.768085] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7282.771133] Lustre: Skipped 2 previous similar messages [ 7288.410977] Lustre: lustre-MDT0001: Client 5f6744fa-5d96-410e-bc2c-258b56e277cb (at 192.168.201.12@tcp) reconnecting [ 7290.447147] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.465979] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7290.476337] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7290.484118] LustreError: 243133:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7290.490225] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7290.494770] LustreError: 243133:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7290.494886] LustreError: 243133:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -108 [ 7290.494903] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7290.494903] [ 7290.497508] LustreError: 239500:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7294.955759] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7296.499195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7302.335210] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7302.578488] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7307.832397] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7307.999487] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7310.580298] Lustre: Permanently deactivating lustre-MDT0001 [ 7310.593463] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7310.598564] Lustre: Skipped 2 previous similar messages [ 7319.156399] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7325.671459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7325.679807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.704366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.926066] Lustre: server umount lustre-MDT0000 complete [ 7330.545778] LustreError: 239490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449150 with bad export cookie 10404810920559811182 [ 7330.566329] LustreError: 239490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7330.566630] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7330.921937] Lustre: server umount lustre-MDT0001 complete [ 7336.824796] Lustre: server umount lustre-OST0000 complete [ 7347.732416] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:26:06 (1788449166) [ 7358.002768] Lustre: server umount lustre-OST0001 complete [ 7364.752614] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7373.423766] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7410.710241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.194694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7411.207035] Lustre: Skipped 2 previous similar messages [ 7411.258226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7411.421646] Lustre: lustre-MDT0000: new disk, initializing [ 7411.578707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.609385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7417.042428] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7427.732273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.906628] Lustre: 247767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7428.063441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7428.073263] Lustre: Skipped 1 previous similar message [ 7428.233938] Lustre: lustre-MDT0001: new disk, initializing [ 7428.386718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.492789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7428.512255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7433.796411] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7444.016471] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7449.174623] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7454.639455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.837550] Lustre: lustre-OST0000: new disk, initializing [ 7454.839689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7454.845188] Lustre: 249736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7454.913769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.908639] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7465.087636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7465.099198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7465.256813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7472.150165] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7473.985651] LustreError: 247774:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7481.942835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7482.130110] Lustre: lustre-OST0001: new disk, initializing [ 7482.133260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7482.142574] Lustre: 250853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7484.079092] LustreError: 247774:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7484.101939] LustreError: 247775:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7488.393236] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7489.110492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7489.133382] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7489.240814] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7494.151295] LustreError: 247775:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7496.975764] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7498.364441] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7502.958522] Lustre: lustre-OST0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 7502.962544] Lustre: Skipped 4 previous similar messages [ 7504.871823] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7504.875104] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.880631] LustreError: Skipped 1 previous similar message [ 7504.890356] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7504.893114] Lustre: Skipped 4 previous similar messages [ 7504.918159] Lustre: Skipped 1 previous similar message [ 7506.504850] Lustre: server umount lustre-OST0001 complete [ 7508.030744] LustreError: 249720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.238394] LustreError: 250208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.242590] LustreError: 250208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7525.360125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.392578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.394871] Lustre: Skipped 1 previous similar message [ 7528.564139] Lustre: server umount lustre-OST0000 complete [ 7535.585487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.597361] Lustre: Skipped 2 previous similar messages [ 7535.606478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7535.626994] Lustre: Skipped 1 previous similar message [ 7538.994525] Lustre: server umount lustre-MDT0000 complete [ 7540.704225] LustreError: 247780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.731575] LustreError: 247780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7542.823696] LustreError: 247759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449362 with bad export cookie 10404810920559813779 [ 7542.826709] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.831994] LustreError: 247759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7549.424661] Lustre: server umount lustre-MDT0001 complete [ 7561.322690] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7564.298099] Key type lgssc unregistered [ 7564.660576] LNet: 252891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.670410] LNetError: 252891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.696996] LNet: Removed LNI 192.168.201.112@tcp [ 7565.623665] Key type .llcrypt unregistered [ 7565.624944] Key type ._llcrypt unregistered [ 7607.155873] Key type ._llcrypt registered [ 7607.164658] Key type .llcrypt registered [ 7607.283933] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7608.329646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7608.441932] alg: No test for adler32 (adler32-zlib) [ 7609.551623] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7609.911961] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7611.591592] Key type lgssc registered [ 7612.793294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7621.283611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7621.303789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7622.568506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7622.590700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7622.724126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.402885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7634.853738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.912611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7634.955879] Lustre: 255100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7635.260533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7639.535832] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7646.002449] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.138883] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7656.994513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.155471] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7657.304394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.086188] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7663.774343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7670.623980] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7725.047101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7725.054629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.075439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.275331] Lustre: server umount lustre-OST0000 complete [ 7731.995936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7745.973721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.591361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.093977] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7761.403992] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7764.972690] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7764.995370] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7765.003246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7800.804456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.809321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.811391] Lustre: Skipped 1 previous similar message [ 7800.816878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.824933] Lustre: Skipped 2 previous similar messages [ 7805.926631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7805.936824] Lustre: Skipped 1 previous similar message [ 7806.638326] Lustre: server umount lustre-OST0000 complete [ 7811.047151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.055187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7811.066637] Lustre: Skipped 1 previous similar message [ 7811.086926] Lustre: Skipped 1 previous similar message [ 7816.161350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.168447] Lustre: Skipped 1 previous similar message [ 7824.351134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7824.938480] Lustre: server umount lustre-MDT0000 complete [ 7828.644108] LustreError: 255094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449648 with bad export cookie 9795919522458523416 [ 7828.659643] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.918895] Lustre: server umount lustre-MDT0001 complete [ 7839.056590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7841.629191] Key type lgssc unregistered [ 7841.895862] LNet: 260528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7841.904878] LNetError: 260528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7841.927506] LNet: Removed LNI 192.168.201.112@tcp [ 7842.643613] Key type .llcrypt unregistered [ 7842.645232] Key type ._llcrypt unregistered [ 7867.559481] Key type ._llcrypt registered [ 7867.560760] Key type .llcrypt registered [ 7867.724351] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7869.560307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.633172] alg: No test for adler32 (adler32-zlib) [ 7870.760921] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7871.128324] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7872.874100] Key type lgssc registered [ 7874.218774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7881.548248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7881.579043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.142712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.026620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7893.479374] LustreError: 261944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7893.492434] LustreError: 261944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7898.596686] LustreError: 261945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.043979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.722700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7907.132587] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7917.465127] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7925.240537] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7934.157751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.492960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.995353] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7945.741458] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7952.948099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7978.471801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.478903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7978.490531] Lustre: Skipped 1 previous similar message [ 7978.511294] Lustre: Skipped 1 previous similar message [ 7983.600312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.610931] Lustre: Skipped 1 previous similar message [ 7984.111336] Lustre: server umount lustre-OST0000 complete [ 7989.733535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7989.743857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.781676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.827715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.849124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.796118] Lustre: server umount lustre-MDT0000 complete [ 7998.945468] LustreError: 261944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.958981] LustreError: 261944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7999.835346] LustreError: 261925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449819 with bad export cookie 17316537765501076064 [ 7999.845560] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7999.846647] LustreError: 261925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8000.469086] Lustre: server umount lustre-MDT0001 complete [ 8013.101119] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8016.852743] Key type lgssc unregistered [ 8017.307164] LNet: 266279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8017.313296] LNetError: 266279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8017.334404] LNet: Removed LNI 192.168.201.112@tcp [ 8018.700190] Key type .llcrypt unregistered [ 8018.705443] Key type ._llcrypt unregistered [ 8038.415594] Key type ._llcrypt registered [ 8038.420188] Key type .llcrypt registered [ 8038.526270] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8039.596371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8039.662563] alg: No test for adler32 (adler32-zlib) [ 8040.828512] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8040.987852] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8042.623210] Key type lgssc registered [ 8043.660436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8051.365859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8051.400288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.925878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.312925] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8062.506991] LustreError: 267501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.525068] LustreError: 267501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8066.466465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.133444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8073.096033] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8082.788857] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8088.544700] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8095.187454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8095.473325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.554371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8102.439077] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8110.137689] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8123.370098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.371746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8123.411441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8123.425593] Lustre: Skipped 1 previous similar message [ 8128.481692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8128.492803] Lustre: Skipped 1 previous similar message [ 8129.452853] Lustre: server umount lustre-OST0000 complete [ 8133.605936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8133.607266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.615901] Lustre: Skipped 1 previous similar message [ 8133.620674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8133.625844] Lustre: Skipped 1 previous similar message [ 8138.721786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8138.730804] Lustre: Skipped 1 previous similar message [ 8141.920846] Lustre: server umount lustre-MDT0000 complete [ 8143.842169] LustreError: 267500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.860763] LustreError: 267500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8145.864216] LustreError: 267483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449965 with bad export cookie 3041669300677430907 [ 8145.870541] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.147898] Lustre: server umount lustre-MDT0001 complete [ 8159.256978] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8162.810183] Key type lgssc unregistered [ 8163.132830] LNet: 271185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8163.142988] LNetError: 271185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8164.203662] LNet: Removed LNI 192.168.201.112@tcp [ 8165.135645] Key type .llcrypt unregistered [ 8165.138096] Key type ._llcrypt unregistered [ 8192.082058] Key type ._llcrypt registered [ 8192.085208] Key type .llcrypt registered [ 8192.209859] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8193.406989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8193.430512] alg: No test for adler32 (adler32-zlib) [ 8194.738644] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8195.014083] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8196.695211] Key type lgssc registered [ 8198.418624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.683243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.704704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.375568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.036909] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8219.620408] LustreError: 272607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.634067] LustreError: 272607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8224.742620] LustreError: 272608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8224.923241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.544397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8231.040691] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8238.893226] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8245.242576] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8251.879616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.217292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.289741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8259.412349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8267.623924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8295.397756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.402583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.418779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.423048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8296.425658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.431186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.446728] Lustre: Skipped 1 previous similar message [ 8300.763431] Lustre: server umount lustre-OST0000 complete [ 8306.658074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.661548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8306.672513] Lustre: Skipped 1 previous similar message [ 8306.687337] Lustre: Skipped 1 previous similar message [ 8310.716517] Lustre: server umount lustre-MDT0000 complete [ 8311.777163] LustreError: 272602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.799896] LustreError: 272602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8314.645933] LustreError: 272590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450134 with bad export cookie 11804888436070168343 [ 8314.668518] LustreError: 272590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8314.670346] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.063232] Lustre: server umount lustre-MDT0001 complete [ 8325.773849] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8328.549667] Key type lgssc unregistered [ 8328.780976] LNet: 276915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8328.788488] LNetError: 276915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8329.841292] LNet: Removed LNI 192.168.201.112@tcp [ 8330.776240] Key type .llcrypt unregistered [ 8330.778240] Key type ._llcrypt unregistered [ 8348.928803] Key type ._llcrypt registered [ 8348.931232] Key type .llcrypt registered [ 8349.067519] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8350.296363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8350.453666] alg: No test for adler32 (adler32-zlib) [ 8351.700949] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8351.894917] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8353.543750] Key type lgssc registered [ 8354.554980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8363.163953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8363.178595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.716256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.282431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8374.244799] LustreError: 278152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.256388] LustreError: 278152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8377.238989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.586790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.190091] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8387.502966] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8392.968319] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.573076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.056333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.276546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8409.308887] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8417.936760] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8430.066584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8430.073253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8430.084329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.099276] Lustre: Skipped 1 previous similar message [ 8434.746495] Lustre: server umount lustre-OST0000 complete [ 8439.265017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8439.271595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.290115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8439.295171] Lustre: Skipped 1 previous similar message [ 8440.291452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8440.294953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.410228] LustreError: 278144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8445.432745] LustreError: 278144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8446.478460] Lustre: server umount lustre-MDT0000 complete [ 8450.329283] LustreError: 278117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450270 with bad export cookie 9241902930601255647 [ 8450.344429] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8450.347443] LustreError: 278117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8450.733941] Lustre: server umount lustre-MDT0001 complete [ 8463.737143] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8467.013408] Key type lgssc unregistered [ 8467.286056] LNet: 281855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8467.294580] LNetError: 281855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8467.320348] LNet: Removed LNI 192.168.201.112@tcp [ 8468.161916] Key type .llcrypt unregistered [ 8468.164415] Key type ._llcrypt unregistered [ 8480.537693] Key type ._llcrypt registered [ 8480.539821] Key type .llcrypt registered [ 8480.619283] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8497.828991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8512.363328] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8513.534363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8513.582208] alg: No test for adler32 (adler32-zlib) [ 8514.874668] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8515.210272] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8516.960699] Key type lgssc registered [ 8518.989206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.095852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8554.136580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.494693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8555.520065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8555.662228] Lustre: lustre-MDT0000: new disk, initializing [ 8555.775284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.784890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8560.286128] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8571.411072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.531428] Lustre: 285409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8571.615910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8571.620458] Lustre: Skipped 1 previous similar message [ 8571.766812] Lustre: lustre-MDT0001: new disk, initializing [ 8571.915244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8571.960432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8571.977849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8577.498498] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8586.806260] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8592.183631] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8599.498740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.868482] Lustre: lustre-OST0000: new disk, initializing [ 8599.871533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8599.877666] Lustre: 287381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8600.007271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.801339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8605.812816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8605.958152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8606.823448] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8615.759780] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8620.543327] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8620.812356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8625.481282] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8625.709591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8631.783300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.787489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8631.794982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8631.804254] LustreError: Skipped 1 previous similar message [ 8634.118319] Lustre: server umount lustre-OST0000 complete [ 8638.446776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8638.452103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.469139] Lustre: Skipped 1 previous similar message [ 8638.473802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8638.479629] Lustre: Skipped 1 previous similar message [ 8642.021240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.023188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.944626] Lustre: server umount lustre-MDT0000 complete [ 8647.635026] LustreError: 286205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450467 with bad export cookie 4535726137621495208 [ 8647.638976] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.645404] LustreError: 286205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8647.971374] Lustre: server umount lustre-MDT0001 complete [ 8655.470632] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:47:54 (1788450474) [ 8661.381203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.999680] LustreError: 289790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.016746] LustreError: 289790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8662.068187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.829865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8667.112854] LustreError: 289791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.133725] LustreError: 289791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8672.224857] LustreError: 289790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.262361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.838102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8682.084670] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8689.214898] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8694.149167] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8700.371376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8700.564738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.129239] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8711.773778] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8727.526167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8727.543119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.563421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8727.566483] Lustre: Skipped 1 previous similar message [ 8729.748923] Lustre: server umount lustre-OST0000 complete [ 8738.272710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8738.285475] LustreError: Skipped 1 previous similar message [ 8738.292544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.309007] Lustre: Skipped 1 previous similar message [ 8738.315956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.318365] Lustre: Skipped 2 previous similar messages [ 8740.029864] Lustre: server umount lustre-MDT0000 complete [ 8743.909695] LustreError: 292088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.941131] LustreError: 292088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8744.007910] LustreError: 289772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450564 with bad export cookie 4535726137621496090 [ 8744.018681] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8744.392198] Lustre: server umount lustre-MDT0001 complete [ 8754.594419] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8756.743626] Key type lgssc unregistered [ 8756.913694] LNet: 293330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8756.918167] LNetError: 293330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8756.937221] LNet: Removed LNI 192.168.201.112@tcp [ 8757.604358] Key type .llcrypt unregistered [ 8757.613498] Key type ._llcrypt unregistered [ 8783.276690] Key type ._llcrypt registered [ 8783.280274] Key type .llcrypt registered [ 8783.377078] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8797.463253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8798.714483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8798.742804] alg: No test for adler32 (adler32-zlib) [ 8799.902214] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8800.201860] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8801.895174] Key type lgssc registered [ 8803.063409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.617524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.637683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.910061] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8850.934642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8851.051953] Lustre: lustre-MDT0000: new disk, initializing [ 8851.171164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.191569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8855.702052] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8866.130781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.212938] Lustre: 297158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8866.262714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8866.264887] Lustre: Skipped 1 previous similar message [ 8866.319818] Lustre: lustre-MDT0001: new disk, initializing [ 8866.404722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8866.435734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8866.445899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8870.869980] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8879.799919] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8885.907516] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8893.037913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8893.374938] Lustre: lustre-OST2710: new disk, initializing [ 8893.377496] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8893.381642] Lustre: 299126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8893.454811] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8899.073599] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8902.223564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8902.227067] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8902.386140] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8908.110734] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8915.870088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8916.139498] Lustre: lustre-OST03e8: new disk, initializing [ 8916.144950] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8916.152406] Lustre: 300191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8916.265618] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8921.802054] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8923.787566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8923.800994] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8923.929900] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8929.207562] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8936.292969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8936.484928] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8940.648960] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8940.921966] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8945.623827] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8945.849622] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8951.080671] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8951.346854] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8958.128797] Lustre: server umount lustre-MDT0000 complete [ 8958.431876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8958.436565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8959.981220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8960.946485] LustreError: 297952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450781 with bad export cookie 5697365926423542045 [ 8960.948100] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8960.962633] LustreError: 297952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8961.410293] Lustre: server umount lustre-MDT0001 complete [ 8975.624743] Lustre: server umount lustre-OST2710 complete [ 8990.843475] Lustre: server umount lustre-OST03e8 complete [ 9009.328668] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 9019.873946] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9052.637215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.879897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9052.908973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9052.979489] Lustre: lustre-MDT0000: new disk, initializing [ 9053.067610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.088694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9058.840140] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9071.611627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9071.736178] Lustre: 305438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9071.778595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9071.781889] Lustre: Skipped 1 previous similar message [ 9071.859289] Lustre: lustre-MDT0001: new disk, initializing [ 9071.968695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.004127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9072.017218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9077.327301] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9087.182477] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9092.360432] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9099.216219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.468497] Lustre: lustre-OST0000: new disk, initializing [ 9099.475393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9099.482899] Lustre: 307405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9099.539104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9104.948049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9104.959719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9105.043056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9105.247480] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9114.085276] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9118.292845] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9118.460255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9123.345853] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9123.582901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9125.857183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9125.867305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.880172] Lustre: Skipped 2 previous similar messages [ 9125.895350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9130.469864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9130.482981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.494436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9131.959588] Lustre: server umount lustre-OST0000 complete [ 9136.100337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.104662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9136.118512] Lustre: Skipped 1 previous similar message [ 9136.140712] Lustre: Skipped 2 previous similar messages [ 9141.219820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9141.226384] Lustre: Skipped 1 previous similar message [ 9141.833880] Lustre: server umount lustre-MDT0000 complete [ 9144.921285] LustreError: 307404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450964 with bad export cookie 5697365926423543725 [ 9144.928899] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9144.943765] LustreError: 307404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9145.289884] Lustre: server umount lustre-MDT0001 complete [ 9153.377705] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:56:12 (1788450972) [ 9154.867469] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9156.473457] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:56:15 (1788450975) [ 9160.244349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9182.919657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.084123] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9183.095728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9183.198495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.640365] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9196.040523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.143821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9196.204699] Lustre: 311265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9201.202588] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9221.688411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9221.878890] Lustre: Denying initial registration attempt from nid 192.168.201.112@tcp, specified as failover [ 9221.886412] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9221.893426] LustreError: 312718:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9221.898462] LustreError: 312718:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9221.902381] LustreError: 312718:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9221.976482] Lustre: server umount lustre-OST0000 complete [ 9221.983408] LustreError: 312718:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9232.352472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9232.365192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.366887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9232.383150] Lustre: Skipped 1 previous similar message [ 9232.395374] Lustre: Skipped 1 previous similar message [ 9234.308449] Lustre: server umount lustre-MDT0000 complete [ 9237.473897] LustreError: 311272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9237.502602] LustreError: 311272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9237.913598] LustreError: 312017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451057 with bad export cookie 5697365926423544607 [ 9237.917314] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9237.922837] LustreError: 312017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9238.350508] Lustre: server umount lustre-MDT0001 complete [ 9249.298969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9252.131103] Key type lgssc unregistered [ 9252.462981] LNet: 313843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9252.471429] LNetError: 313843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9253.547556] LNet: Removed LNI 192.168.201.112@tcp [ 9254.710072] Key type .llcrypt unregistered [ 9254.714625] Key type ._llcrypt unregistered [ 9266.717601] Key type ._llcrypt registered [ 9266.721217] Key type .llcrypt registered [ 9266.973432] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9267.849562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9267.858316] alg: No test for adler32 (adler32-zlib) [ 9268.895111] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9269.141569] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9270.896239] Key type lgssc registered [ 9273.071762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9319.013646] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9327.683722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9327.706459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.958229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9328.998384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9329.201931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.839170] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9342.464162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.534931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9342.571576] Lustre: 316604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9342.735291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9342.882501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9348.222508] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9356.044964] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9362.379456] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9382.574954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9382.735678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9382.883270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.536376] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9396.084050] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9400.291809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9400.296164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.297890] LustreError: Skipped 1 previous similar message [ 9400.306816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9400.312842] Lustre: Skipped 1 previous similar message [ 9400.333771] Lustre: Skipped 1 previous similar message [ 9403.916892] Lustre: server umount lustre-OST0000 complete [ 9409.505734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9409.510398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.519965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.534765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.780868] Lustre: server umount lustre-MDT0000 complete [ 9417.334958] LustreError: 316597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451237 with bad export cookie 11550212281077214666 [ 9417.337539] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9417.350484] LustreError: 316597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9417.636872] Lustre: server umount lustre-MDT0001 complete [ 9427.432125] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9430.275754] Key type lgssc unregistered [ 9430.683874] LNet: 320183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9430.691165] LNetError: 320183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9430.713837] LNet: Removed LNI 192.168.201.112@tcp [ 9432.042475] Key type .llcrypt unregistered [ 9432.044136] Key type ._llcrypt unregistered [ 9460.765721] Key type ._llcrypt registered [ 9460.769919] Key type .llcrypt registered [ 9460.886825] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9462.310877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9462.420286] alg: No test for adler32 (adler32-zlib) [ 9463.710448] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9464.038419] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9465.807948] Key type lgssc registered [ 9467.181818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9476.111861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9476.142094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.586104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.906578] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9487.846369] LustreError: 321613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.869417] LustreError: 321613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9491.965683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.459846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9497.955383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9505.998154] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9512.294868] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9521.909245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9522.152485] LustreError: 323437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9522.371410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9529.757676] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9538.619232] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9547.232624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.237027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.247194] Lustre: Skipped 2 previous similar messages [ 9547.257450] Lustre: Skipped 2 previous similar messages [ 9550.434309] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 9552.357903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.369389] Lustre: Skipped 2 previous similar messages [ 9552.655126] Lustre: server umount lustre-MDT0000 complete [ 9555.514536] LustreError: 321607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.467447] LustreError: 322348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451376 with bad export cookie 2052417175550894378 [ 9556.484155] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9556.487539] LustreError: 322348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9557.475240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.496953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9560.639694] LustreError: 324058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.654414] LustreError: 324058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9561.505165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9561.508749] Lustre: Skipped 1 previous similar message [ 9562.998283] Lustre: server umount lustre-MDT0001 complete [ 9569.766504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9577.909814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.026213] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1c7ba68c079f3e96 [ 9581.032661] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [ 9581.265434] LustreError: 323447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9581.286958] LustreError: 323447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9581.328207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.393636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9586.101373] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9591.595956] LustreError: 323448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9591.604970] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9591.607018] LustreError: 323448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9593.598545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.900429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9598.427946] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9598.957093] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9605.488808] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9607.092626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9612.073124] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9613.578350] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9619.436100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.446793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9619.450610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9624.311722] Lustre: server umount lustre-OST0000 complete [ 9629.664486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.672594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.678585] Lustre: Skipped 2 previous similar messages [ 9634.005227] Lustre: server umount lustre-MDT0000 complete [ 9634.797359] LustreError: 325208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.819564] LustreError: 325208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9637.847635] LustreError: 321592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451457 with bad export cookie 2052417175550901910 [ 9637.859151] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9637.862679] LustreError: 321592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9638.191143] Lustre: server umount lustre-MDT0001 complete [ 9649.170992] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9651.936462] Key type lgssc unregistered [ 9652.300141] LNet: 327880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9652.307299] LNetError: 327880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9652.330158] LNet: Removed LNI 192.168.201.112@tcp [ 9653.000938] Key type .llcrypt unregistered [ 9653.002703] Key type ._llcrypt unregistered [ 9677.014495] Key type ._llcrypt registered [ 9677.020786] Key type .llcrypt registered [ 9677.152865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9678.728681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9678.818540] alg: No test for adler32 (adler32-zlib) [ 9680.145585] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9680.407407] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9682.119220] Key type lgssc registered [ 9683.395867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9692.203707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9692.229606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.764433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.374739] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9703.908048] LustreError: 329298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.920274] LustreError: 329298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9707.395098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.010702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9711.894650] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9719.667535] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9726.787868] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9731.942598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.954224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.969131] Lustre: Skipped 1 previous similar message [ 9731.992613] Lustre: Skipped 1 previous similar message [ 9737.194923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9737.208042] Lustre: Skipped 1 previous similar message [ 9742.305578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.319084] Lustre: Skipped 1 previous similar message [ 9745.375336] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9745.540152] Lustre: server umount lustre-MDT0000 complete [ 9747.430330] LustreError: 329298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.458556] LustreError: 329298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9749.728316] LustreError: 329280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451569 with bad export cookie 523769468142215651 [ 9749.737328] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9749.749503] LustreError: 329280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9750.125504] Lustre: server umount lustre-MDT0001 complete [ 9766.483905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.788282] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9766.820578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9767.075426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.504330] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9779.771518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.880391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9779.918453] Lustre: 331890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9780.208473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9784.969764] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9792.952381] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9799.140590] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9809.145313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9809.378892] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9809.391157] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9809.402994] LustreError: 333774:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9809.411140] LustreError: 333774:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9809.422123] LustreError: 333774:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9809.598460] Lustre: server umount lustre-OST0000 complete [ 9809.601725] LustreError: 333774:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9818.679336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9818.875075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9819.089243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.176269] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9828.961182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9833.729588] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9842.251295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9842.368541] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9842.462135] Lustre: lustre-OST0001: new disk, initializing [ 9842.465512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9842.471150] Lustre: 335317:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9842.610153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9849.400377] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9850.483750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9850.504466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9850.585742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9857.555265] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9861.094057] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.094791] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9861.096616] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9861.119297] Lustre: Skipped 1 previous similar message [ 9866.210777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9866.240308] Lustre: Skipped 2 previous similar messages [ 9866.817833] Lustre: server umount lustre-OST0001 complete [ 9871.329759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.335387] LustreError: 334241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.371661] LustreError: 334241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9876.453725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9876.456514] LustreError: 334242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9876.465495] Lustre: Skipped 3 previous similar messages [ 9876.519327] LustreError: 334242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9881.571654] LustreError: 334240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9881.604842] LustreError: 334240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9885.151292] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9885.298925] Lustre: server umount lustre-OST0000 complete [ 9891.811793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.832328] Lustre: Skipped 2 previous similar messages [ 9896.366354] Lustre: server umount lustre-MDT0000 complete [ 9896.930836] LustreError: 331895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.963627] LustreError: 331895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9900.837635] LustreError: 331882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451720 with bad export cookie 523769468142216197 [ 9900.840682] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9900.845870] LustreError: 331882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9901.288544] Lustre: server umount lustre-MDT0001 complete [ 9914.193947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9917.691806] Key type lgssc unregistered [ 9918.070307] LNet: 337282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9918.075765] LNetError: 337282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.106707] LNet: Removed LNI 192.168.201.112@tcp [ 9918.946831] Key type .llcrypt unregistered [ 9918.951740] Key type ._llcrypt unregistered [ 9994.488686] Key type ._llcrypt registered [ 9994.491790] Key type .llcrypt registered [ 9994.614438] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [10011.405589] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10013.262538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10013.340240] alg: No test for adler32 (adler32-zlib) [10014.512321] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10014.949845] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10016.615293] Key type lgssc registered [10017.743278] Lustre: Echo OBD driver; http://www.lustre.org/ [10055.161781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10055.182929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.447796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10056.477437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10056.557275] Lustre: lustre-MDT0000: new disk, initializing [10056.651387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10056.671232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10062.060515] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10074.310836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.485459] Lustre: 342215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10074.578277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10074.583918] Lustre: Skipped 1 previous similar message [10074.710895] Lustre: lustre-MDT0001: new disk, initializing [10074.860200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.942848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10074.963732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10080.620378] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10091.346641] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10097.018180] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10104.174199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10104.501303] Lustre: lustre-OST0000: new disk, initializing [10104.504404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10104.510293] Lustre: 344187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10104.602618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10111.798387] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10112.089616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10112.103863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10112.263042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10120.376185] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10125.580765] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10125.856077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10131.011977] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10131.185172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10138.081602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.084758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10138.114214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10139.473727] Lustre: server umount lustre-OST0000 complete [10146.787707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10146.793565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.800085] Lustre: Skipped 1 previous similar message [10146.803358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10146.805805] Lustre: Skipped 1 previous similar message [10148.320615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10148.321083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10148.337857] Lustre: Skipped 1 previous similar message [10149.874118] Lustre: server umount lustre-MDT0000 complete [10153.247789] LustreError: 343020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451973 with bad export cookie 6777920838832207073 [10153.251216] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10153.258681] LustreError: 343020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10153.534729] Lustre: server umount lustre-MDT0001 complete [10162.351845] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:13:01 (1788451981) [10169.729748] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:13:08 (1788451988) [10177.727323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.379794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10182.912657] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10183.653019] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.679955] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10188.770773] LustreError: 346942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.516714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.832848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.282284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10205.439850] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10211.988764] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10218.658705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10219.035221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10225.319964] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10233.206394] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10244.734509] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10246.384278] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10247.936834] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10249.536705] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10251.510299] Lustre: Failing over lustre-MDT0000 [10251.750079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.761124] LustreError: 346938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.777847] LustreError: 346938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10251.842483] Lustre: server umount lustre-MDT0000 complete [10256.507243] LustreError: 346938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.560136] LustreError: 347899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.596484] LustreError: 347899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10270.749615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10270.826968] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10271.151930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10271.201126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10271.810125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10276.344041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10276.397883] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10276.526042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10276.679273] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10286.531558] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10288.212766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10292.352605] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10294.294299] Lustre: Failing over lustre-MDT0000 [10294.615174] Lustre: server umount lustre-MDT0000 complete [10296.803528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.808469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10296.814600] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10296.814611] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10296.828516] Lustre: Skipped 3 previous similar messages [10312.161232] Lustre: 340442:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452116/real 1788452116] req@0000000006ab0385 x1875327671428480/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788452132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10312.190421] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10313.869436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.281395] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10317.302288] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [10322.710483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.753570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10323.043116] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10327.470757] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10328.063314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10328.079183] Lustre: Skipped 2 previous similar messages [10328.111720] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10328.162735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10337.508453] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10339.006658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10348.513539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.518489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10348.537223] Lustre: Skipped 3 previous similar messages [10348.552256] Lustre: Skipped 2 previous similar messages [10351.888087] Lustre: server umount lustre-MDT0000 complete [10353.643641] LustreError: 346937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.663157] LustreError: 346937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10355.557950] LustreError: 346924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452175 with bad export cookie 6777920838832209789 [10355.580642] LustreError: 346924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10355.585319] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.892190] Lustre: server umount lustre-MDT0001 complete [10369.907919] Lustre: server umount lustre-OST0000 complete [10383.775105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10389.620632] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10399.274314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.743184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10399.747327] Lustre: Skipped 1 previous similar message [10406.154580] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10414.477754] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10421.017039] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10427.970514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10429.252716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10435.148483] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10441.137837] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10455.010271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10455.015608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.020358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10455.041445] Lustre: Skipped 1 previous similar message [10455.062069] Lustre: Skipped 1 previous similar message [10457.642117] Lustre: server umount lustre-OST0000 complete [10465.252631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10465.270907] Lustre: Skipped 1 previous similar message [10468.378492] Lustre: server umount lustre-MDT0000 complete [10470.369041] LustreError: 353303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10470.397500] LustreError: 353303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10472.410614] LustreError: 353283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452292 with bad export cookie 6777920838832210538 [10472.421026] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10472.788881] Lustre: server umount lustre-MDT0001 complete [10484.939869] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10488.087352] Key type lgssc unregistered [10488.381739] LNet: 356840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10488.388910] LNetError: 356840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10488.400851] LNet: Removed LNI 192.168.201.112@tcp [10489.166070] Key type .llcrypt unregistered [10489.167708] Key type ._llcrypt unregistered [10515.306227] Key type ._llcrypt registered [10515.308115] Key type .llcrypt registered [10515.401074] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10516.618887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10516.652716] alg: No test for adler32 (adler32-zlib) [10517.733859] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10517.952994] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10519.647141] Key type lgssc registered [10520.642199] Lustre: Echo OBD driver; http://www.lustre.org/ [10529.969640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10530.017640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.854684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10538.214311] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10541.027533] LustreError: 358359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.082804] LustreError: 358359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10546.149319] LustreError: 358358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.249504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.523379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10552.981609] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10561.224996] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10566.475667] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10572.332940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.720132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10576.828770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10579.880390] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10586.783290] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10597.345955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10597.351222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10597.367578] Lustre: Skipped 1 previous similar message [10602.261165] Lustre: server umount lustre-MDT0000 complete [10602.466426] LustreError: 358355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.475850] LustreError: 358355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10605.739453] LustreError: 358339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452425 with bad export cookie 18231441439337360988 [10605.743864] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10605.749875] LustreError: 358339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10606.148488] Lustre: server umount lustre-MDT0001 complete [10619.208065] Lustre: server umount lustre-OST0000 complete [10634.773331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.033793] LustreError: 362070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.102386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10640.362192] LustreError: 362071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.670620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10651.620258] LustreError: 362070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.636916] LustreError: 362070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10652.654558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10653.282823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10657.997405] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10665.477047] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10669.946105] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10676.811313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10677.175602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10681.283518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10683.213896] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10690.839336] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10704.352464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10704.364661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10704.384441] Lustre: Skipped 1 previous similar message [10704.389703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10704.394417] Lustre: Skipped 2 previous similar messages [10706.913611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10706.915842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.922241] Lustre: Skipped 1 previous similar message [10706.933785] Lustre: Skipped 1 previous similar message [10709.087312] Lustre: server umount lustre-MDT0000 complete [10712.042165] LustreError: 362067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10712.068691] LustreError: 362067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10712.807884] LustreError: 362053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452532 with bad export cookie 18231441439337362101 [10712.818152] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10712.821439] LustreError: 362053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10713.349521] Lustre: server umount lustre-MDT0001 complete [10728.018592] Lustre: server umount lustre-OST0000 complete [10744.672059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.366221] LustreError: 365985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.420274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10750.020433] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10759.484562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.677719] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10772.103863] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10777.564747] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10784.185364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10784.445730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10784.457147] Lustre: Skipped 1 previous similar message [10790.608995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10790.673574] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10798.445456] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10805.812744] Lustre: server umount lustre-OST0000 complete [10811.366941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.376257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10811.386826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10811.399826] Lustre: Skipped 1 previous similar message [10816.482836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10816.492030] Lustre: Skipped 1 previous similar message [10819.665842] Lustre: server umount lustre-MDT0000 complete [10821.600593] LustreError: 368477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.634752] LustreError: 368477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10822.626525] Lustre: 357668:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452626/real 1788452626] req@00000000a70c2031 x1875328198860800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788452642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10822.627331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10822.662975] Lustre: 357668:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10822.679955] Lustre: Skipped 1 previous similar message [10823.354540] LustreError: 365967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452643 with bad export cookie 18231441439337363123 [10823.363858] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.828636] Lustre: server umount lustre-MDT0001 complete [10834.802626] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10837.229494] Key type lgssc unregistered [10837.522776] LNet: 369519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10837.531795] LNetError: 369519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10837.562649] LNet: Removed LNI 192.168.201.112@tcp [10838.545847] Key type .llcrypt unregistered [10838.547151] Key type ._llcrypt unregistered [10865.524953] Key type ._llcrypt registered [10865.526844] Key type .llcrypt registered [10865.717886] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10866.842917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10866.927881] alg: No test for adler32 (adler32-zlib) [10868.242486] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10868.509462] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10870.183189] Key type lgssc registered [10871.695511] Lustre: Echo OBD driver; http://www.lustre.org/ [10881.017817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10881.031755] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10881.036090] LustreError: 370959:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10881.054176] LustreError: 370959:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10881.061163] LustreError: 370959:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10881.068405] LustreError: 370959:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10881.074654] LustreError: 370959:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10889.598160] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.608103] LustreError: 371356:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10889.626984] LustreError: 371356:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10889.632825] LustreError: 371356:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10889.637371] LustreError: 371356:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10889.643366] LustreError: 371356:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10909.387305] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10912.439714] Key type lgssc unregistered [10912.698889] LNet: 372364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10912.703620] LNetError: 372364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10912.716813] LNet: Removed LNI 192.168.201.112@tcp [10913.665682] Key type .llcrypt unregistered [10913.670685] Key type ._llcrypt unregistered [10930.904511] Key type ._llcrypt registered [10930.908584] Key type .llcrypt registered [10931.021788] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [10946.150799] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10947.148557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10947.296573] alg: No test for adler32 (adler32-zlib) [10948.593973] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10948.839653] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10950.527650] Key type lgssc registered [10951.916981] Lustre: Echo OBD driver; http://www.lustre.org/ [10993.425499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10993.440598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.937281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10995.014405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10995.193522] Lustre: lustre-MDT0000: new disk, initializing [10995.350914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10995.370491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11000.150386] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11010.618980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.778504] Lustre: 375521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11010.827918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11010.831842] Lustre: Skipped 1 previous similar message [11010.891855] Lustre: lustre-MDT0001: new disk, initializing [11011.011502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.068936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11011.119926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11016.222240] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11027.940260] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11034.685235] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11044.188397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.453115] Lustre: lustre-OST0000: new disk, initializing [11044.560185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11044.587829] Lustre: 377489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11044.740603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11050.163271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11050.168801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11050.269725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11052.652590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11063.640393] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11067.242903] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11067.360053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11073.767482] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11073.984736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11076.074429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.086892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11076.100610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11076.109401] Lustre: Skipped 1 previous similar message [11081.195404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11086.333579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11086.348402] Lustre: Skipped 2 previous similar messages [11090.399191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11090.785959] Lustre: server umount lustre-OST0000 complete [11096.549641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11096.555243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11096.562966] Lustre: Skipped 2 previous similar messages [11096.567977] Lustre: Skipped 1 previous similar message [11101.667737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11101.674405] Lustre: Skipped 1 previous similar message [11109.344810] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11109.603768] Lustre: server umount lustre-MDT0000 complete [11114.118608] LustreError: 377488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452934 with bad export cookie 4511601147199180075 [11114.131905] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11114.134673] LustreError: 377488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11114.624607] Lustre: server umount lustre-MDT0001 complete [11125.792288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11128.060455] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:29:06 (1788452946) [11135.582433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.282238] LustreError: 379954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.322460] LustreError: 379954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11136.385472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11141.605138] LustreError: 379955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11142.806688] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11146.722193] LustreError: 379954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11151.847224] LustreError: 379955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.758093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.394261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.512347] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11172.110738] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11179.951500] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11188.364453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.865211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11196.441262] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11205.345963] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11213.351046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11213.612239] Lustre: lustre-OST0001: new disk, initializing [11213.616294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11213.620823] Lustre: 382795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11213.740271] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11221.133586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11221.137701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11221.455129] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11226.646749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11232.286864] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11236.838650] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11236.842072] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11236.904106] Lustre: Skipped 1 previous similar message [11236.906033] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11236.923782] LustreError: Skipped 1 previous similar message [11236.954377] Lustre: Skipped 3 previous similar messages [11242.944889] Lustre: server umount lustre-OST0001 complete [11244.125470] LustreError: 382706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.192715] LustreError: 381779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.224347] LustreError: 381779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11262.435547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.452199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11262.461665] Lustre: Skipped 1 previous similar message [11262.478716] Lustre: Skipped 4 previous similar messages [11268.684824] Lustre: server umount lustre-OST0000 complete [11277.283982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11277.293413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11279.720229] Lustre: server umount lustre-MDT0000 complete [11282.923489] LustreError: 381052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11282.954859] LustreError: 381052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11284.410309] LustreError: 380690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453104 with bad export cookie 4511601147199180957 [11284.426764] LustreError: 380690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11284.428709] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11290.977071] Lustre: server umount lustre-MDT0001 complete [11306.917804] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11312.022891] Key type lgssc unregistered [11312.239646] LNet: 384787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11312.242708] LNetError: 384787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11312.252847] LNet: Removed LNI 192.168.201.112@tcp [11313.498150] Key type .llcrypt unregistered [11313.500544] Key type ._llcrypt unregistered [11382.958835] Key type ._llcrypt registered [11382.961778] Key type .llcrypt registered [11383.119228] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11384.830105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11384.913351] alg: No test for adler32 (adler32-zlib) [11386.061258] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11386.331646] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11388.103193] Key type lgssc registered [11389.529750] Lustre: Echo OBD driver; http://www.lustre.org/ [11399.288424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11399.308031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.721652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11400.747450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11400.979436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11407.320595] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11417.524703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11417.675679] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11417.716541] Lustre: 387592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11417.950817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11424.258717] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11434.194428] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11441.649150] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11444.196396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.215882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11444.218045] Lustre: Skipped 1 previous similar message [11444.249269] Lustre: Skipped 1 previous similar message [11449.312374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.329213] Lustre: Skipped 1 previous similar message [11450.278050] Lustre: server umount lustre-MDT0000 complete [11454.496563] LustreError: 388344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453274 with bad export cookie 7415817544458703408 [11454.507540] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11454.781683] Lustre: server umount lustre-MDT0001 complete [11466.490342] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:34:44 (1788453284) [11474.841385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.512517] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.533040] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11475.608650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11481.062065] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.833450] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11486.188897] LustreError: 390106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.299821] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.486784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.061095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11497.959222] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11507.452075] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11514.114734] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11521.788659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11522.042296] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11522.358835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.099741] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11538.450380] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11547.986473] Lustre: Permanently deactivating lustre-OST0000 [11547.998760] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11557.448728] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11560.647148] LustreError: 393143:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11560.658791] LustreError: 393143:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11560.665040] LustreError: 393143:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11564.654125] LustreError: 393191:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11564.662713] LustreError: 393191:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11573.054967] Lustre: server umount lustre-OST0000 complete [11576.391562] LustreError: 393441:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11576.399630] LustreError: 393441:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11582.433750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.434844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11582.437788] Lustre: Skipped 1 previous similar message [11587.556086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11587.563961] Lustre: Skipped 2 previous similar messages [11589.688840] Lustre: server umount lustre-MDT0000 complete [11592.673848] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.713535] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11594.500238] LustreError: 390085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453414 with bad export cookie 7415817544458703989 [11594.510073] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11594.520204] LustreError: 390085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11604.136902] Lustre: server umount lustre-MDT0001 complete [11615.427894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11621.666388] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11625.318979] LustreError: 394603:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11633.812405] Lustre: 394942:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.112@tcp [11636.155649] Lustre: server umount MGS complete [11649.158809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.690175] LustreError: 395666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11649.714839] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11649.720396] Lustre: Skipped 1 previous similar message [11649.794635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11655.242512] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11664.239211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.664163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11669.812443] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11678.191438] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11684.817930] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11686.979896] Lustre: Permanently reactivating lustre-OST0000 [11686.994582] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11686.999745] Lustre: Skipped 2 previous similar messages [11715.040069] LustreError: 397449:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11724.437958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.046720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.054420] Lustre: Skipped 1 previous similar message [11732.887779] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11742.533716] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11749.868932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11749.894260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.924661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11750.882968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11761.125659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11761.151457] Lustre: Skipped 4 previous similar messages [11764.191126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11764.554831] Lustre: server umount lustre-OST0000 complete [11771.366272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11771.388679] Lustre: Skipped 2 previous similar messages [11776.420939] Lustre: server umount lustre-MDT0000 complete [11776.483976] LustreError: 395662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.513753] LustreError: 395662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11781.354756] LustreError: 395648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453601 with bad export cookie 7415817544458705039 [11781.367605] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11781.378055] LustreError: 395648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11781.873212] Lustre: server umount lustre-MDT0001 complete [11796.877861] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11800.205965] Key type lgssc unregistered [11800.563974] LNet: 399989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11800.569394] LNetError: 399989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11800.580779] LNet: Removed LNI 192.168.201.112@tcp [11801.880148] Key type .llcrypt unregistered [11801.881942] Key type ._llcrypt unregistered [11823.255683] Key type ._llcrypt registered [11823.259955] Key type .llcrypt registered [11823.421432] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [11841.088739] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11842.996224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11843.008538] alg: No test for adler32 (adler32-zlib) [11844.097614] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11844.425639] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11846.153500] Key type lgssc registered [11847.965354] Lustre: Echo OBD driver; http://www.lustre.org/ [11886.686450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11886.709335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.110045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11888.162313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11888.292682] Lustre: lustre-MDT0000: new disk, initializing [11888.460136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11888.487892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11893.189278] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11905.433840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11905.626049] Lustre: 403144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11905.709668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11905.712283] Lustre: Skipped 1 previous similar message [11905.844630] Lustre: lustre-MDT0001: new disk, initializing [11906.038727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11906.101646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11906.128793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11912.331931] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11922.394347] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11928.967959] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11936.214968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11936.633143] Lustre: lustre-OST0000: new disk, initializing [11936.641601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11936.655630] Lustre: 405117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11936.790595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11943.503160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11943.519883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11943.725542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11944.040033] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11953.812708] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11959.513884] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11959.760959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11965.378172] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11965.641639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11969.505106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11969.515675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.522189] LustreError: Skipped 1 previous similar message [11969.529173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11969.550309] Lustre: Skipped 1 previous similar message [11969.569800] Lustre: Skipped 1 previous similar message [11974.633987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11974.642571] Lustre: Skipped 1 previous similar message [11979.745083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.756319] Lustre: Skipped 1 previous similar message [11982.303148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11982.546089] Lustre: server umount lustre-OST0000 complete [11987.941102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11987.952468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11987.960570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11989.986722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.123156] Lustre: server umount lustre-MDT0000 complete [11997.609460] LustreError: 403138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453817 with bad export cookie 3522435594851955085 [11997.615581] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11997.629337] LustreError: 403138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11998.001932] Lustre: server umount lustre-MDT0001 complete [12007.747667] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12009.911450] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:43:48 (1788453828) [12017.724299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.191449] LustreError: 407579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.206305] LustreError: 407579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12018.278302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12022.920930] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12023.782948] LustreError: 407580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.899093] LustreError: 407579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.644023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.956845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12036.766253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12043.978038] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12049.248470] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12055.829081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12056.175347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12061.856848] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12068.953662] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12071.618255] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12074.720252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12083.171854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12083.176675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12083.191919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12083.205479] Lustre: Skipped 2 previous similar messages [12086.278719] Lustre: server umount lustre-OST0000 complete [12093.409635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12093.414917] LustreError: Skipped 1 previous similar message [12093.421359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12093.443385] Lustre: Skipped 1 previous similar message [12093.450929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12093.454573] Lustre: Skipped 2 previous similar messages [12104.159230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12104.438221] Lustre: server umount lustre-MDT0000 complete [12104.674763] LustreError: 407579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12104.687628] LustreError: 407579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12107.944076] LustreError: 407560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453928 with bad export cookie 3522435594851955967 [12107.966309] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12108.408840] Lustre: server umount lustre-MDT0001 complete [12117.219602] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12119.961835] Key type lgssc unregistered [12120.335112] LNet: 411309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12120.347086] LNetError: 411309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12121.387586] LNet: Removed LNI 192.168.201.112@tcp [12122.720106] Key type .llcrypt unregistered [12122.722115] Key type ._llcrypt unregistered [12148.029938] Key type ._llcrypt registered [12148.032620] Key type .llcrypt registered [12148.155276] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12148.490482] Key type .llcrypt unregistered [12148.492090] Key type ._llcrypt unregistered [12164.576508] Key type ._llcrypt registered [12164.582046] Key type .llcrypt registered [12164.674521] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12166.028398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12166.061768] alg: No test for adler32 (adler32-zlib) [12167.176941] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12167.400732] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12169.015171] Key type lgssc registered [12169.811766] Lustre: Echo OBD driver; http://www.lustre.org/ [12177.532317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12177.560961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.330788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12184.092895] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12189.676507] LustreError: 413528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.696880] LustreError: 413528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12192.759506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12193.023755] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.067120] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12193.084417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12198.370823] LustreError: 413528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.397510] LustreError: 413528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12199.132536] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12203.489546] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.507584] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12204.528115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12205.584687] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12209.641574] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12209.665614] LustreError: 413547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12214.306203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.839879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12221.491431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12235.155144] Lustre: server umount lustre-OST0000 complete [12238.818196] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12238.821357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12238.826777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12240.352654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12240.355142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.476109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.489043] Lustre: Skipped 2 previous similar messages [12248.790049] Lustre: server umount lustre-MDT0000 complete [12250.613683] LustreError: 413525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.632210] LustreError: 413525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12251.615660] Lustre: 412839:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454055/real 1788454055] req@0000000052ea8c39 x1875329927901184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788454071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12251.663231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.159578] LustreError: 413508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454074 with bad export cookie 4628304368486077354 [12254.168409] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12254.176101] LustreError: 413508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12254.617624] Lustre: server umount lustre-MDT0001 complete [12266.718674] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12269.617940] Key type lgssc unregistered [12269.963724] LNet: 416680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12269.968925] LNetError: 416680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12271.017356] LNet: Removed LNI 192.168.201.112@tcp [12272.072748] Key type .llcrypt unregistered [12272.075532] Key type ._llcrypt unregistered [12301.827957] Key type ._llcrypt registered [12301.831547] Key type .llcrypt registered [12301.995500] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12304.136776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12304.238078] alg: No test for adler32 (adler32-zlib) [12305.455456] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12305.812666] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12307.639214] Key type lgssc registered [12309.379967] Lustre: Echo OBD driver; http://www.lustre.org/ [12319.404914] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12319.416510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12335.008268] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12340.130591] LustreError: 418087:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [12366.815239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12373.955939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12382.510691] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12391.609966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12392.548124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12392.668844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12397.539467] LustreError: 418112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.553358] LustreError: 418112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12398.878530] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12401.645168] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.667523] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12406.760290] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12406.784245] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12409.704235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.061614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12415.421720] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12423.543635] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12430.649972] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12435.941560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12435.945756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12435.963633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12435.983913] Lustre: Skipped 1 previous similar message [12441.069121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12441.075746] Lustre: Skipped 1 previous similar message [12442.077603] Lustre: server umount lustre-OST0000 complete [12451.301961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.302873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12451.312961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12451.314563] Lustre: Skipped 1 previous similar message [12451.342609] Lustre: Skipped 2 previous similar messages [12452.795111] Lustre: server umount lustre-MDT0000 complete [12456.421665] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.448094] LustreError: 419124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12456.731794] LustreError: 418095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454276 with bad export cookie 15011270193754526823 [12456.738473] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12456.746057] LustreError: 418095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12457.400956] Lustre: server umount lustre-MDT0001 complete [12470.523537] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12473.649872] Key type lgssc unregistered [12473.987941] LNet: 421661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12474.000286] LNetError: 421661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12475.045371] LNet: Removed LNI 192.168.201.112@tcp [12475.931432] Key type .llcrypt unregistered [12475.934738] Key type ._llcrypt unregistered [12502.293360] Key type ._llcrypt registered [12502.296753] Key type .llcrypt registered [12502.469323] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12504.475990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12504.504368] alg: No test for adler32 (adler32-zlib) [12505.717988] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12506.160354] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12507.911621] Key type lgssc registered [12509.253212] Lustre: Echo OBD driver; http://www.lustre.org/ [12517.110488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12517.135734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.458295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12523.400256] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12528.611424] LustreError: 423078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.664585] LustreError: 423078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12533.735019] LustreError: 423079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.111594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.586275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12540.438489] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12548.005462] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12553.930132] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12561.996800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12562.397332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12563.448376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12569.763885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12578.175566] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12584.931894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.936303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.956037] Lustre: Skipped 2 previous similar messages [12584.967512] Lustre: Skipped 2 previous similar messages [12586.061378] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [12589.855425] Lustre: server umount lustre-MDT0000 complete [12590.056521] LustreError: 423074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.079272] LustreError: 423074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12592.552428] LustreError: 423059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.12@tcp arrived at 1788454412 with bad export cookie 4503622996809073716 [12592.570635] LustreError: 423059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12594.505961] LustreError: 423058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454414 with bad export cookie 4503622996809072827 [12594.514355] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12595.168619] LustreError: 424381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.172681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12595.173942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.179802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12595.193287] LustreError: 424381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12599.203868] LustreError: 423807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.205438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12599.217386] LustreError: 423807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12601.027656] Lustre: server umount lustre-OST0000 complete [12608.470647] Lustre: server umount lustre-MDT0001 complete [12620.937884] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12624.033830] Key type lgssc unregistered [12624.296674] LNet: 426772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12624.303714] LNetError: 426772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12624.323777] LNet: Removed LNI 192.168.201.112@tcp [12625.177407] Key type .llcrypt unregistered [12625.179756] Key type ._llcrypt unregistered [12650.082743] Key type ._llcrypt registered [12650.091780] Key type .llcrypt registered [12650.216290] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12651.830504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12651.969179] alg: No test for adler32 (adler32-zlib) [12653.017729] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12653.238786] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12654.938316] Key type lgssc registered [12655.873117] Lustre: Echo OBD driver; http://www.lustre.org/ [12662.720201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12662.734970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12664.263538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12669.523747] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12674.535340] LustreError: 428190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12674.569865] LustreError: 428190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12679.649537] LustreError: 428191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.086508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12680.643969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12685.843626] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12693.596872] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12700.001618] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12707.638443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12707.992250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12709.063687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12715.338970] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12724.055779] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12730.950072] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [12731.876663] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.879161] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12731.903303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12731.918078] Lustre: Skipped 1 previous similar message [12734.534798] Lustre: server umount lustre-MDT0001 complete [12736.060514] LustreError: 428186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.066726] LustreError: 430693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.085822] LustreError: 430693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12745.187675] LustreError: 428929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12745.188883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12745.221615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.228585] Lustre: Skipped 1 previous similar message [12745.238836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12747.612053] Lustre: server umount lustre-OST0000 complete [12751.659166] Lustre: server umount lustre-MDT0000 complete [12767.136950] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12770.827633] Key type lgssc unregistered [12771.132992] LNet: 431877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12771.140311] LNetError: 431877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12772.206276] LNet: Removed LNI 192.168.201.112@tcp [12773.050515] Key type .llcrypt unregistered [12773.053513] Key type ._llcrypt unregistered [12798.533774] Key type ._llcrypt registered [12798.537748] Key type .llcrypt registered [12798.674449] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [12815.885365] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12817.515809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12817.546423] alg: No test for adler32 (adler32-zlib) [12818.699123] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12819.029445] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12820.791163] Key type lgssc registered [12822.064809] Lustre: Echo OBD driver; http://www.lustre.org/ [12863.111882] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12880.234459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12880.275694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12881.638449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12881.660715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12881.754110] Lustre: lustre-MDT0000: new disk, initializing [12881.837034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12881.860301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12887.119767] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12902.396352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12902.599693] Lustre: lustre-OST0000: new disk, initializing [12902.603458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12902.606642] Lustre: Skipped 1 previous similar message [12902.610802] Lustre: 436914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12902.663158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12908.068897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12908.078510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12908.117901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12908.118947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12917.092361] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12933.230900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12933.392834] Lustre: 435894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12933.499960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12933.695195] Lustre: lustre-MDT0001: new disk, initializing [12933.872313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12933.948971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12933.963312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12938.545822] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12947.420807] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12947.652959] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12951.574397] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12951.799345] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12956.249288] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [12960.224819] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12960.227691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12960.243178] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12960.285594] Lustre: Skipped 1 previous similar message [12961.341490] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [12961.357628] Lustre: Skipped 1 previous similar message [12962.930578] Lustre: server umount lustre-MDT0001 complete [12970.467428] LustreError: 435907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12970.479424] LustreError: 435907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12975.584869] LustreError: 438238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.606732] LustreError: 438238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12980.716254] LustreError: 435906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.727363] LustreError: 435906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12984.810811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12984.824125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12984.842369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12985.836726] LustreError: 435907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12988.510329] Lustre: server umount lustre-OST0000 complete [12990.954684] LustreError: 435906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.327531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13007.606621] Lustre: server umount lustre-MDT0000 complete [13022.284484] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13025.371943] Key type lgssc unregistered [13025.780736] LNet: 440666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13025.790634] LNetError: 440666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13025.823724] LNet: Removed LNI 192.168.201.112@tcp [13026.827690] Key type .llcrypt unregistered [13026.834028] Key type ._llcrypt unregistered [13070.326485] Key type ._llcrypt registered [13070.328310] Key type .llcrypt registered [13070.450185] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13070.713329] Key type .llcrypt unregistered [13070.722769] Key type ._llcrypt unregistered [13087.802706] Key type ._llcrypt registered [13087.804531] Key type .llcrypt registered [13087.883805] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13089.330263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13089.338386] alg: No test for adler32 (adler32-zlib) [13090.707809] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13091.037834] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13092.752755] Key type lgssc registered [13093.950251] Lustre: Echo OBD driver; http://www.lustre.org/ [13129.573430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13129.591956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13130.723774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13130.740270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13130.785415] Lustre: lustre-MDT0000: new disk, initializing [13130.828496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13130.839194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13134.142458] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13146.155099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.298447] Lustre: 444061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13146.395559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13146.400712] Lustre: Skipped 1 previous similar message [13146.475500] Lustre: lustre-MDT0001: new disk, initializing [13146.539499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13146.581824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13146.592016] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13152.068989] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13163.104953] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13169.411713] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13177.153306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13177.623487] Lustre: lustre-OST0000: new disk, initializing [13177.627900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13177.642073] Lustre: 446029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13177.812631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13183.938674] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13186.151827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13186.169656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13186.286920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13195.536344] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13202.914168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13202.922888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13202.943337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13207.014875] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13207.017331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13207.042254] Lustre: Skipped 1 previous similar message [13208.612226] Lustre: server umount lustre-MDT0000 complete [13212.479325] LustreError: 446028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455032 with bad export cookie 5011704763938694117 [13212.501161] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13212.506167] LustreError: 446028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13212.911910] Lustre: server umount lustre-MDT0001 complete [13227.632277] Lustre: server umount lustre-OST0000 complete [13240.193381] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:04:18 (1788455058) [13249.521596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13250.297783] LustreError: 448064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13250.329340] LustreError: 448064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13250.411365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13255.655745] LustreError: 448065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13256.000939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13260.779472] LustreError: 448064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13265.605585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13265.864410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13271.335935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13279.993447] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13285.414192] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13305.777448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13306.051644] Lustre: Found index 0 for lustre-OST0000, updating log [13306.334057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13314.620833] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13321.717779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13321.996663] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13325.792944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13325.798923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13325.813565] Lustre: Skipped 1 previous similar message [13325.819881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13331.649826] Lustre: server umount lustre-OST0000 complete [13352.929966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13352.945223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13352.961329] Lustre: Skipped 1 previous similar message [13352.980288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13352.988215] Lustre: Skipped 3 previous similar messages [13354.864597] Lustre: server umount lustre-MDT0000 complete [13355.489656] LustreError: 449016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13355.515361] LustreError: 449016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13359.514053] LustreError: 448045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455179 with bad export cookie 5011704763938695510 [13359.519355] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13360.028948] Lustre: server umount lustre-MDT0001 complete [13368.607462] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [13369.253915] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:06:28 (1788455188) [13996.473152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13996.813159] LustreError: 454480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13996.853613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14001.203918] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14002.148426] LustreError: 454481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14006.248387] LustreError: 454480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14009.624254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.898330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14014.405827] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14023.281988] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14030.216208] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14051.241373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14051.618061] Lustre: Found index 0 for lustre-OST0000, updating log [14051.902469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14059.223316] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14059.872414] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [14067.892330] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14071.077330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14071.100882] Lustre: Skipped 1 previous similar message [14071.109962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14071.124823] Lustre: Skipped 4 previous similar messages [14076.518422] Lustre: server umount lustre-OST0000 complete [14081.511306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14081.521580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14081.528921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14081.550929] Lustre: Skipped 1 previous similar message [14081.581689] Lustre: Skipped 1 previous similar message [14082.599840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14084.712419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14084.719552] Lustre: Skipped 4 previous similar messages [14086.875639] Lustre: server umount lustre-MDT0000 complete [14087.781978] LustreError: 454476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14087.796454] LustreError: 454476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14090.794581] LustreError: 454461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455910 with bad export cookie 5011704763938696595 [14090.808753] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14090.824606] LustreError: 454461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14091.516789] Lustre: server umount lustre-MDT0001 complete [14123.359234] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [14124.560750] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [14134.901084] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14175.914514] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:19:54 (1788455994) [14197.782845] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [14200.909611] Key type lgssc unregistered [14201.262458] LNet: 461097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14201.271273] LNetError: 461097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14312.935099] LNet: Removed LNI 192.168.201.112@tcp [14314.153484] Key type .llcrypt unregistered [14314.154929] Key type ._llcrypt unregistered [14332.481760] Key type ._llcrypt registered [14332.483662] Key type .llcrypt registered [14332.578173] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14333.705266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14333.806755] alg: No test for adler32 (adler32-zlib) [14334.722880] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14334.994924] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [14336.695262] Key type lgssc registered [14338.100638] Lustre: Echo OBD driver; http://www.lustre.org/ [14342.750361] LNet: Added LNI 192.168.201.112@tcp100 [8/256/0/180] [14343.855866] LNet: Added LNI 192.168.201.112@tcp101 [8/256/0/180] [14345.253100] LNet: Added LNI 192.168.201.112@tcp102 [8/256/0/180] [14346.692928] LNet: Added LNI 192.168.201.112@tcp103 [8/256/0/180] [14348.230760] LNet: Added LNI 192.168.201.112@tcp104 [8/256/0/180] [14350.199903] LNet: Added LNI 192.168.201.112@tcp105 [8/256/0/180] [14351.470891] LNet: Added LNI 192.168.201.112@tcp106 [8/256/0/180] [14353.032813] LNet: Added LNI 192.168.201.112@tcp107 [8/256/0/180] [14354.681262] LNet: Added LNI 192.168.201.112@tcp108 [8/256/0/180] [14356.073984] LNet: Added LNI 192.168.201.112@tcp109 [8/256/0/180] [14357.684485] LNet: Added LNI 192.168.201.112@tcp110 [8/256/0/180] [14359.365796] LNet: Added LNI 192.168.201.112@tcp111 [8/256/0/180] [14360.386065] LNet: Added LNI 192.168.201.112@tcp112 [8/256/0/180] [14361.736677] LNet: Added LNI 192.168.201.112@tcp113 [8/256/0/180] [14362.990476] LNet: Added LNI 192.168.201.112@tcp114 [8/256/0/180] [14364.760899] LNet: Added LNI 192.168.201.112@tcp115 [8/256/0/180] [14366.301082] LNet: Added LNI 192.168.201.112@tcp116 [8/256/0/180] [14367.622735] LNet: Added LNI 192.168.201.112@tcp117 [8/256/0/180] [14369.294850] LNet: Added LNI 192.168.201.112@tcp118 [8/256/0/180] [14370.263300] LNet: Added LNI 192.168.201.112@tcp119 [8/256/0/180] [14371.312175] LNet: Added LNI 192.168.201.112@tcp120 [8/256/0/180] [14372.358374] LNet: Added LNI 192.168.201.112@tcp121 [8/256/0/180] [14373.283191] LNet: Added LNI 192.168.201.112@tcp122 [8/256/0/180] [14374.335847] LNet: Added LNI 192.168.201.112@tcp123 [8/256/0/180] [14375.565045] LNet: Added LNI 192.168.201.112@tcp124 [8/256/0/180] [14376.611952] LNet: Added LNI 192.168.201.112@tcp125 [8/256/0/180] [14377.528330] LNet: Added LNI 192.168.201.112@tcp126 [8/256/0/180] [14378.632994] LNet: Added LNI 192.168.201.112@tcp127 [8/256/0/180] [14379.720172] LNet: Added LNI 192.168.201.112@tcp128 [8/256/0/180] [14380.660793] LNet: Added LNI 192.168.201.112@tcp129 [8/256/0/180] [14381.803921] LNet: Added LNI 192.168.201.112@tcp130 [8/256/0/180] [14382.751023] LNet: Added LNI 192.168.201.112@tcp131 [8/256/0/180] [14383.704231] LNet: Added LNI 192.168.201.112@tcp132 [8/256/0/180] [14384.677744] LNet: Added LNI 192.168.201.112@tcp133 [8/256/0/180] [14385.533910] LNet: Added LNI 192.168.201.112@tcp134 [8/256/0/180] [14386.692135] LNet: Added LNI 192.168.201.112@tcp135 [8/256/0/180] [14395.244722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14395.296927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14396.525355] Lustre: 464129:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14396.645507] LNetError: 461587:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14396.664906] LNetError: Protocol error connecting to 192.168.201.112@tcp116 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14396.808561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14396.871584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14397.002949] Lustre: lustre-MDT0000: new disk, initializing [14397.119920] Lustre: 464187:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14397.146414] Lustre: 464187:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14397.175536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14397.212312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14397.728193] LNetError: 464164:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14397.738836] LNetError: 464164:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14397.748070] LNetError: Protocol error connecting to 192.168.201.112@tcp at host 192.168.201.112:988: is it running a compatible version of Lustre? [14397.756763] LNetError: Skipped 4 previous similar messages [14401.924244] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14413.855372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14414.051138] Lustre: 464146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14414.209389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14414.222481] Lustre: Skipped 1 previous similar message [14414.395254] Lustre: lustre-MDT0001: new disk, initializing [14414.489377] Lustre: 464950:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14414.496499] Lustre: 464950:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14414.637715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14414.665716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14414.692036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14419.266717] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14424.142444] Lustre: 465375:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14424.159197] Lustre: 465375:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14426.302186] LNetError: 464163:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14426.311675] LNetError: Protocol error connecting to 192.168.201.112@tcp at host 192.168.201.112:988: is it running a compatible version of Lustre? [14431.085819] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14431.465106] LNetError: 464157:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14431.482173] LNetError: 464157:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14431.487662] LNetError: Protocol error connecting to 192.168.201.112@tcp121 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14431.499675] LNetError: Skipped 1 previous similar message [14435.560648] LNetError: 461589:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14435.584532] LNetError: 461589:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14435.600910] LNetError: Protocol error connecting to 192.168.201.112@tcp122 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14435.619737] LNetError: Skipped 2 previous similar messages [14437.124295] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14443.756283] LNetError: 464149:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14443.767494] LNetError: 464149:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14443.779313] LNetError: Protocol error connecting to 192.168.201.112@tcp121 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14443.794742] LNetError: Skipped 2 previous similar messages [14443.975053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14444.524449] Lustre: lustre-OST0000: new disk, initializing [14444.528651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14444.545729] Lustre: 466123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14444.673538] Lustre: 466128:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14444.679918] Lustre: 466128:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14444.740382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14452.021750] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14454.315157] Lustre: 466580:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [14454.327273] Lustre: 466580:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14454.389610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14454.397629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14454.537348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14461.154540] LNetError: 461589:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14461.172907] LNetError: 461589:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [14461.181704] LNetError: Protocol error connecting to 192.168.201.112@tcp122 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14461.194501] LNetError: Skipped 8 previous similar messages [14461.475589] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14470.112334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14470.115171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14470.120328] LustreError: Skipped 1 previous similar message [14470.125849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14470.148742] Lustre: Skipped 1 previous similar message [14475.301626] Lustre: server umount lustre-OST0000 complete [14480.356750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14480.370916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14480.380590] Lustre: Skipped 1 previous similar message [14480.395161] Lustre: Skipped 2 previous similar messages [14485.473522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14485.483666] Lustre: Skipped 1 previous similar message [14486.197922] Lustre: server umount lustre-MDT0000 complete [14490.897988] LustreError: 464137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456310 with bad export cookie 7042459423218872978 [14490.908380] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14491.166495] Lustre: server umount lustre-MDT0001 complete [14495.976870] LNetError: 464152:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [14495.986781] LNetError: 464152:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14495.998362] LNetError: Protocol error connecting to 192.168.201.112@tcp121 at host 192.168.201.112:988: is it running a compatible version of Lustre? [14496.008605] LNetError: Skipped 7 previous similar messages [14503.592582] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [14506.939370] Key type lgssc unregistered [14507.571828] LNet: 468042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14507.578325] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14508.641207] LNet: Removed LNI 192.168.201.112@tcp [14508.671901] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14508.687193] LNet: Removed LNI 192.168.201.112@tcp100 [14509.727234] LNet: Removed LNI 192.168.201.112@tcp101 [14509.747518] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14509.756848] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14509.767119] LNet: Removed LNI 192.168.201.112@tcp102 [14510.817364] LNet: Removed LNI 192.168.201.112@tcp103 [14511.839507] LNet: Removed LNI 192.168.201.112@tcp104 [14511.853095] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14511.862785] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14511.874656] LNet: Removed LNI 192.168.201.112@tcp105 [14512.927220] LNet: Removed LNI 192.168.201.112@tcp106 [14513.951206] LNet: Removed LNI 192.168.201.112@tcp107 [14514.975314] LNet: Removed LNI 192.168.201.112@tcp108 [14516.000226] LNet: Removed LNI 192.168.201.112@tcp109 [14516.005141] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14516.011821] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14516.022836] LNet: Removed LNI 192.168.201.112@tcp110 [14517.087200] LNet: Removed LNI 192.168.201.112@tcp111 [14518.111244] LNet: Removed LNI 192.168.201.112@tcp112 [14519.135201] LNet: Removed LNI 192.168.201.112@tcp113 [14520.159284] LNet: Removed LNI 192.168.201.112@tcp114 [14521.183340] LNet: Removed LNI 192.168.201.112@tcp115 [14522.207165] LNet: Removed LNI 192.168.201.112@tcp116 [14523.231181] LNet: Removed LNI 192.168.201.112@tcp117 [14524.255301] LNet: Removed LNI 192.168.201.112@tcp118 [14524.258554] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14524.270283] LNetError: 468042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14524.285458] LNet: Removed LNI 192.168.201.112@tcp119 [14525.343538] LNet: Removed LNI 192.168.201.112@tcp120 [14526.369185] LNet: Removed LNI 192.168.201.112@tcp121 [14527.455147] LNet: Removed LNI 192.168.201.112@tcp122 [14528.482051] LNet: Removed LNI 192.168.201.112@tcp123 [14528.495761] LNet: Removed LNI 192.168.201.112@tcp124 [14528.499998] LNet: Removed LNI 192.168.201.112@tcp125 [14528.505117] LNet: Removed LNI 192.168.201.112@tcp126 [14528.509192] LNet: Removed LNI 192.168.201.112@tcp127 [14528.512415] LNet: Removed LNI 192.168.201.112@tcp128 [14528.526072] LNet: Removed LNI 192.168.201.112@tcp129 [14528.533309] LNet: Removed LNI 192.168.201.112@tcp130 [14528.545848] LNet: Removed LNI 192.168.201.112@tcp131 [14528.547896] LNet: Removed LNI 192.168.201.112@tcp132 [14528.550897] LNet: Removed LNI 192.168.201.112@tcp133 [14528.565154] LNet: Removed LNI 192.168.201.112@tcp134 [14528.585469] LNet: Removed LNI 192.168.201.112@tcp135 [14529.685212] Key type .llcrypt unregistered [14529.686615] Key type ._llcrypt unregistered [14555.503634] Key type ._llcrypt registered [14555.510496] Key type .llcrypt registered [14555.591775] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [14575.828499] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14578.422123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14578.526229] alg: No test for adler32 (adler32-zlib) [14579.854348] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14580.347411] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [14582.143174] Key type lgssc registered [14583.091175] Lustre: Echo OBD driver; http://www.lustre.org/ [14629.659931] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:27:27 (1788456447) [14654.447643] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [14657.429820] Key type lgssc unregistered [14657.739935] LNet: 472138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14657.745908] LNetError: 472138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14657.761971] LNet: Removed LNI 192.168.201.112@tcp [14659.094381] Key type .llcrypt unregistered [14659.098076] Key type ._llcrypt unregistered [14679.664714] Key type ._llcrypt registered [14679.667338] Key type .llcrypt registered [14679.815533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14681.556690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14681.601923] alg: No test for adler32 (adler32-zlib) [14682.800063] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14683.204182] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [14684.929494] Key type lgssc registered [14686.229249] Lustre: Echo OBD driver; http://www.lustre.org/ [14697.528789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14697.585085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14699.025664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14699.114362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14699.249299] Lustre: lustre-MDT0000: new disk, initializing [14699.363434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14699.383831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14705.343957] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14717.098035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14717.242544] Lustre: 473352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14717.293590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14717.295804] Lustre: Skipped 1 previous similar message [14717.354746] Lustre: lustre-MDT0001: new disk, initializing [14717.463878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14717.498751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14717.506393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14722.124318] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14731.593674] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14736.834564] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14743.591691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14743.848539] Lustre: lustre-OST0000: new disk, initializing [14743.855922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14743.860978] Lustre: 475320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14743.971774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14749.328100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14749.348509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14749.556046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14750.214328] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14761.048886] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14767.630486] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing lnet_if_list [14777.348751] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14777.386302] LNet: Added LNI 192.168.201.112@tcp10 [8/256/0/180] [14786.210895] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14786.273521] LNet: Added LNI 192.168.201.112@tcp11 [8/256/0/180] [14791.798777] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14791.889781] LNet: Added LNI 192.168.201.112@tcp12 [8/256/0/180] [14796.971702] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14796.992317] LNet: Added LNI 192.168.201.112@tcp13 [8/256/0/180] [14802.125387] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14802.186310] LNet: Added LNI 192.168.201.112@tcp14 [8/256/0/180] [14807.269482] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14807.296391] LNet: Added LNI 192.168.201.112@tcp15 [8/256/0/180] [14810.532784] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14810.553634] LNet: Added LNI 192.168.201.112@tcp16 [8/256/0/180] [14816.859531] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14816.905758] LNet: Added LNI 192.168.201.112@tcp17 [8/256/0/180] [14824.774172] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14824.802148] LNet: Added LNI 192.168.201.112@tcp18 [8/256/0/180] [14829.969589] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14829.987411] LNet: Added LNI 192.168.201.112@tcp19 [8/256/0/180] [14835.113797] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14835.162582] LNet: Added LNI 192.168.201.112@tcp20 [8/256/0/180] [14840.516082] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14840.540067] LNet: Added LNI 192.168.201.112@tcp21 [8/256/0/180] [14846.799803] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14846.826609] LNet: Added LNI 192.168.201.112@tcp22 [8/256/0/180] [14853.474628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14853.524528] LNet: Added LNI 192.168.201.112@tcp23 [8/256/0/180] [14859.732626] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14859.780425] LNet: Added LNI 192.168.201.112@tcp24 [8/256/0/180] [14867.031444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14867.057640] LNet: Added LNI 192.168.201.112@tcp25 [8/256/0/180] [14875.090207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14875.158092] LNet: Added LNI 192.168.201.112@tcp26 [8/256/0/180] [14882.046984] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14882.067062] LNet: Added LNI 192.168.201.112@tcp27 [8/256/0/180] [14888.766222] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14888.794448] LNet: Added LNI 192.168.201.112@tcp28 [8/256/0/180] [14895.735382] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14895.784957] LNet: Added LNI 192.168.201.112@tcp29 [8/256/0/180] [14904.008622] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14904.049728] LNet: Added LNI 192.168.201.112@tcp30 [8/256/0/180] [14912.693095] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14912.722435] LNet: Added LNI 192.168.201.112@tcp31 [8/256/0/180] [14921.810371] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14921.841446] LNet: Added LNI 192.168.201.112@tcp32 [8/256/0/180] [14930.198768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14930.244792] LNet: Added LNI 192.168.201.112@tcp33 [8/256/0/180] [14938.270516] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14938.302031] LNet: Added LNI 192.168.201.112@tcp34 [8/256/0/180] [14946.936293] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14946.960628] LNet: Added LNI 192.168.201.112@tcp35 [8/256/0/180] [14954.908826] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14954.932617] LNet: Added LNI 192.168.201.112@tcp36 [8/256/0/180] [14963.243204] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14963.286156] LNet: Added LNI 192.168.201.112@tcp37 [8/256/0/180] [14971.799178] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14971.875349] LNet: Added LNI 192.168.201.112@tcp38 [8/256/0/180] [14980.607579] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14980.660420] LNet: Added LNI 192.168.201.112@tcp39 [8/256/0/180] [14990.180938] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14990.223711] LNet: Added LNI 192.168.201.112@tcp40 [8/256/0/180] [14998.044481] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14998.077459] LNet: Added LNI 192.168.201.112@tcp41 [8/256/0/180] [15008.817383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15008.861653] LNet: Added LNI 192.168.201.112@tcp42 [8/256/0/180] [15022.506225] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15022.594971] LNet: Added LNI 192.168.201.112@tcp43 [8/256/0/180] [15034.910466] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15034.961025] LNet: Added LNI 192.168.201.112@tcp44 [8/256/0/180] [15043.791962] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15043.838217] LNet: Added LNI 192.168.201.112@tcp45 [8/256/0/180] [15051.551896] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15051.646857] LNet: Added LNI 192.168.201.112@tcp46 [8/256/0/180] [15061.237480] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15061.292518] LNet: Added LNI 192.168.201.112@tcp47 [8/256/0/180] [15071.511435] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15071.588807] LNet: Added LNI 192.168.201.112@tcp48 [8/256/0/180] [15083.684359] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15083.720787] LNet: Added LNI 192.168.201.112@tcp49 [8/256/0/180] [15093.830547] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15093.851563] LNet: Added LNI 192.168.201.112@tcp50 [8/256/0/180] [15116.768401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15116.771575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15116.812540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15118.309818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15118.311467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15120.109307] Lustre: server umount lustre-MDT0000 complete [15124.032423] LustreError: 473344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456944 with bad export cookie 3580086952602435127 [15124.037954] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15124.045423] LustreError: 473344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15124.610616] Lustre: server umount lustre-MDT0001 complete [15144.927362] Lustre: 472625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456949/real 1788456949] req@00000000d431b298 x1875332566280448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15144.928121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15144.968380] Lustre: 472625:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15144.997325] Lustre: Skipped 1 previous similar message [15146.488381] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15149.026110] Lustre: 472623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456953/real 1788456953] req@0000000074e23c5c x1875332566280576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15150.111722] Lustre: 472623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456954/real 1788456954] req@00000000e0b9f75e x1875332566280832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15155.170769] Lustre: 472625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456959/real 1788456959] req@000000000522dbf7 x1875332566280960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15157.175446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15169.568928] LustreError: 472621:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009e70e197 x1875332566281600/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15170.006074] LustreError: 475313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15170.028508] LustreError: 475313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15170.080886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15171.109137] LustreError: 475314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.152630] LustreError: 475314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15173.159994] LustreError: 499732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15173.178309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15174.936944] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15178.212334] LustreError: 499568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15178.236466] LustreError: 499568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15183.332704] LustreError: 499570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15183.353490] LustreError: 499570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15183.389988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15183.589694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15189.001300] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15189.715184] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15200.582875] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15207.229765] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15229.931109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15229.942396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15229.942742] Lustre: Skipped 1 previous similar message [15229.946386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15229.946397] Lustre: Skipped 2 previous similar messages [15233.144369] Lustre: server umount lustre-OST0000 complete [15240.160334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15240.188934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15240.198606] Lustre: Skipped 2 previous similar messages [15244.521238] Lustre: server umount lustre-MDT0000 complete [15245.281151] LustreError: 499569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15245.299134] LustreError: 499569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15248.893480] LustreError: 473342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457068 with bad export cookie 3580086952602437787 [15248.905603] LustreError: 473342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15248.907951] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15249.148372] Lustre: server umount lustre-MDT0001 complete [15262.044631] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15264.996533] Key type lgssc unregistered [15265.327655] LNet: 502368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15265.334038] LNetError: 502368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15265.341329] LNet: Removed LNI 192.168.201.112@tcp [15265.343845] LNet: Removed LNI 192.168.201.112@tcp10 [15265.346277] LNet: Removed LNI 192.168.201.112@tcp11 [15265.348569] LNet: Removed LNI 192.168.201.112@tcp12 [15265.351299] LNet: Removed LNI 192.168.201.112@tcp13 [15265.353474] LNet: Removed LNI 192.168.201.112@tcp14 [15265.355509] LNet: Removed LNI 192.168.201.112@tcp15 [15265.357602] LNet: Removed LNI 192.168.201.112@tcp16 [15265.359622] LNet: Removed LNI 192.168.201.112@tcp17 [15265.361996] LNet: Removed LNI 192.168.201.112@tcp18 [15265.363784] LNet: Removed LNI 192.168.201.112@tcp19 [15265.365611] LNet: Removed LNI 192.168.201.112@tcp20 [15265.367652] LNet: Removed LNI 192.168.201.112@tcp21 [15265.369204] LNet: Removed LNI 192.168.201.112@tcp22 [15265.370781] LNet: Removed LNI 192.168.201.112@tcp23 [15265.372520] LNet: Removed LNI 192.168.201.112@tcp24 [15265.374255] LNet: Removed LNI 192.168.201.112@tcp25 [15265.375674] LNet: Removed LNI 192.168.201.112@tcp26 [15265.377250] LNet: Removed LNI 192.168.201.112@tcp27 [15265.378402] LNet: Removed LNI 192.168.201.112@tcp28 [15265.380021] LNet: Removed LNI 192.168.201.112@tcp29 [15265.381950] LNet: Removed LNI 192.168.201.112@tcp30 [15265.384428] LNet: Removed LNI 192.168.201.112@tcp31 [15265.386385] LNet: Removed LNI 192.168.201.112@tcp32 [15265.389499] LNet: Removed LNI 192.168.201.112@tcp33 [15265.392507] LNet: Removed LNI 192.168.201.112@tcp34 [15265.394834] LNet: Removed LNI 192.168.201.112@tcp35 [15265.406515] LNet: Removed LNI 192.168.201.112@tcp36 [15265.409332] LNet: Removed LNI 192.168.201.112@tcp37 [15265.412458] LNet: Removed LNI 192.168.201.112@tcp38 [15265.414897] LNet: Removed LNI 192.168.201.112@tcp39 [15265.418154] LNet: Removed LNI 192.168.201.112@tcp40 [15265.420637] LNet: Removed LNI 192.168.201.112@tcp41 [15265.423785] LNet: Removed LNI 192.168.201.112@tcp42 [15265.426717] LNet: Removed LNI 192.168.201.112@tcp43 [15265.428962] LNet: Removed LNI 192.168.201.112@tcp44 [15265.433982] LNet: Removed LNI 192.168.201.112@tcp45 [15265.436833] LNet: Removed LNI 192.168.201.112@tcp46 [15265.439358] LNet: Removed LNI 192.168.201.112@tcp47 [15265.442116] LNet: Removed LNI 192.168.201.112@tcp48 [15265.444189] LNet: Removed LNI 192.168.201.112@tcp49 [15265.453218] LNet: Removed LNI 192.168.201.112@tcp50 [15266.440922] Key type .llcrypt unregistered [15266.443225] Key type ._llcrypt unregistered [15289.431643] Key type ._llcrypt registered [15289.434192] Key type .llcrypt registered [15289.548386] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [15306.013545] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15307.607352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15307.637089] alg: No test for adler32 (adler32-zlib) [15308.719941] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15308.951516] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15310.615172] Key type lgssc registered [15312.020616] Lustre: Echo OBD driver; http://www.lustre.org/ [15347.232267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15347.258227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15348.667980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15348.732825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15348.829917] Lustre: lustre-MDT0000: new disk, initializing [15348.946254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15348.960604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15354.899394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15368.179765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15368.340311] Lustre: 505562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15368.431132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15368.436430] Lustre: Skipped 1 previous similar message [15368.546232] Lustre: lustre-MDT0001: new disk, initializing [15368.667311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15368.734763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15368.759502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15372.965796] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15383.123455] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15390.192629] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15399.242123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15399.464470] Lustre: *** cfs_fail_loc=90f, val=0*** [15399.514450] Lustre: lustre-OST0000: new disk, initializing [15399.518686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15399.526306] Lustre: 507579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15399.612177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15407.200255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15407.213156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15407.351483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15407.441326] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15419.382850] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15433.188847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15433.211477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15433.223497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15433.233432] Lustre: Skipped 1 previous similar message [15434.851915] Lustre: server umount lustre-OST0000 complete [15440.352538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15440.358022] LustreError: Skipped 1 previous similar message [15440.366051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15440.382321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15440.395896] Lustre: Skipped 1 previous similar message [15443.425378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15443.426886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15445.689494] Lustre: server umount lustre-MDT0000 complete [15449.487947] LustreError: 505557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457269 with bad export cookie 12245907388436338269 [15449.495936] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15449.812757] Lustre: server umount lustre-MDT0001 complete [15461.454150] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15464.428228] Key type lgssc unregistered [15464.726499] LNet: 509490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15464.742649] LNetError: 509490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15464.786050] LNet: Removed LNI 192.168.201.112@tcp [15465.860950] Key type .llcrypt unregistered [15465.863392] Key type ._llcrypt unregistered [15484.687545] Key type ._llcrypt registered [15484.689456] Key type .llcrypt registered [15484.789900] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15486.489692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15486.512909] alg: No test for adler32 (adler32-zlib) [15487.906273] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15488.109949] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15489.767181] Key type lgssc registered [15490.774710] Lustre: Echo OBD driver; http://www.lustre.org/ [15500.398578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15500.444401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15502.004617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15505.798077] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15512.553035] LustreError: 510764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15512.582604] LustreError: 510764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15517.032687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15517.383707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15522.561601] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15530.951201] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15536.559978] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15541.791549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15542.083932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15549.177646] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15556.578710] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15565.515241] loop0: detected capacity change from 0 to 204800000 [15565.728146] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15565.742340] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15565.757350] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15571.372439] loop0: detected capacity change from 0 to 204800000 [15571.518260] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15571.533090] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15571.562205] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15576.693199] loop0: detected capacity change from 0 to 204800000 [15582.146391] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15582.307745] Lustre: fs15246-OST0000: new disk, initializing [15582.311747] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15582.319827] Lustre: 514357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15582.380809] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15588.494301] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15597.472589] loop1: detected capacity change from 0 to 204800000 [15602.606904] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15602.746456] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15602.815189] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15602.924451] Lustre: fs15246-MDT0000: new disk, initializing [15603.201677] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15603.230546] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15603.383396] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15604.449914] Lustre: 515478:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007fee3639 x1875333410238464/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788457440 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15609.571864] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15609.586655] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15609.647423] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15609.693361] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15633.900082] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.907906] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15639.036971] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15639.969327] Lustre: server umount fs15246-MDT0000 complete [15644.135033] LustreError: 512605:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15649.256255] LustreError: 512605:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15649.879996] Lustre: server umount fs15246-OST0000 complete [15664.616159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15664.636472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15669.739058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15669.747100] Lustre: Skipped 2 previous similar messages [15674.855306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15674.873888] Lustre: Skipped 1 previous similar message [15675.871211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15676.092196] Lustre: server umount lustre-OST0000 complete [15679.969270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15679.996141] Lustre: Skipped 2 previous similar messages [15685.091265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15685.113349] Lustre: Skipped 3 previous similar messages [15685.701977] Lustre: server umount lustre-MDT0000 complete [15689.467808] LustreError: 511498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457509 with bad export cookie 12011442656431367385 [15689.477093] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15689.489730] LustreError: 511498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15690.038777] Lustre: server umount lustre-MDT0001 complete [15701.190379] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15704.284828] Key type lgssc unregistered [15704.577185] LNet: 518581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15704.596471] LNetError: 518581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15704.625587] LNet: Removed LNI 192.168.201.112@tcp [15705.653199] Key type .llcrypt unregistered [15705.655324] Key type ._llcrypt unregistered [15725.905452] Key type ._llcrypt registered [15725.910366] Key type .llcrypt registered [15726.037733] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15727.256868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15727.463180] alg: No test for adler32 (adler32-zlib) [15728.756232] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15729.009425] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15730.743182] Key type lgssc registered [15731.810595] Lustre: Echo OBD driver; http://www.lustre.org/ [15776.540688] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [15790.318365] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15830.743433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15830.767263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15831.973180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15831.991647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15832.061731] Lustre: lustre-MDT0000: new disk, initializing [15832.115471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15832.123103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15836.524312] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15848.590663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15848.786474] Lustre: 523053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15848.934901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15848.938506] Lustre: Skipped 1 previous similar message [15849.027169] Lustre: lustre-MDT0001: new disk, initializing [15849.157715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15849.188908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15849.200725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15856.802992] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15867.760082] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15875.153664] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15883.071829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15883.381566] Lustre: lustre-OST0000: new disk, initializing [15883.384396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15883.391987] Lustre: 525021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15883.502767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15890.127889] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15893.670192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15893.676780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15893.835754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15899.771780] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15904.273493] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15904.615985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15912.174914] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15912.503515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15919.612433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15919.642569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15919.651517] LustreError: Skipped 1 previous similar message [15919.663211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15919.680232] Lustre: Skipped 1 previous similar message [15919.713312] Lustre: Skipped 1 previous similar message [15921.964951] Lustre: server umount lustre-OST0000 complete [15929.863183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15929.865889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15929.908744] Lustre: Skipped 1 previous similar message [15934.947078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15934.963227] Lustre: Skipped 2 previous similar messages [15939.054395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15939.068151] Lustre: Skipped 1 previous similar message [15941.087250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15941.397424] Lustre: server umount lustre-MDT0000 complete [15945.129255] LustreError: 525020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457765 with bad export cookie 8147960330383397980 [15945.137366] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15945.156927] LustreError: 525020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15945.609493] Lustre: server umount lustre-MDT0001 complete [15973.712783] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [15986.337457] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [16023.133249] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15749 sec ========= 13:50:41 (1788457841) [16025.324023] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:50:43 (1788457843) === [16029.371798] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:50:47 (1788457847) === [16076.599629] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [16079.419922] Key type lgssc unregistered [16079.798610] LNet: 531673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16079.806939] LNetError: 531673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16079.829173] LNet: Removed LNI 192.168.201.112@tcp [16080.817443] Key type .llcrypt unregistered [16080.820166] Key type ._llcrypt unregistered