[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468184194 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003085] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012048] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014051] SELinux: Initializing. [ 0.016078] *** VALIDATE selinux *** [ 0.025119] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030061] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032165] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033111] *** VALIDATE tmpfs *** [ 0.035178] *** VALIDATE proc *** [ 0.036227] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.039030] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042031] Spectre V2 : User space: Vulnerable [ 0.043007] Speculative Store Bypass: Vulnerable [ 0.045709] debug: unmapping init [mem 0xffffffff86259000-0xffffffff86260fff] [ 0.047224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048669] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051026] ... generic registers: 4 [ 0.052028] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056294] rcu: Hierarchical SRCU implementation. [ 0.058525] smp: Bringing up secondary CPUs ... [ 0.059617] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.063666] smp: Brought up 1 node, 4 CPUs [ 0.065018] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139107] node 0 deferred pages initialised in 69ms [ 0.142152] devtmpfs: initialized [ 0.143230] x86/mm: Memory block size: 128MB [ 0.145817] gcov: version magic: 0x41383552 [ 0.148321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151223] pinctrl core: initialized pinctrl subsystem [ 0.153194] [ 0.153860] ************************************************************* [ 0.157014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ** ** [ 0.162017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164010] ** ** [ 0.166013] ** This means that this kernel is built to expose internal ** [ 0.168015] ** IOMMU data structures, which may compromise security on ** [ 0.171020] ** your system. ** [ 0.174014] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.178013] ** kernel, report this immediately to your vendor! ** [ 0.181020] ** ** [ 0.183010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ************************************************************* [ 0.187452] NET: Registered protocol family 16 [ 0.189326] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194119] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199028] cpuidle: using governor menu [ 0.200863] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203700] PCI: Using configuration type 1 for base access [ 0.205132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221086] cryptd: max_cpu_qlen set to 1000 [ 0.224223] ACPI: Added _OSI(Module Device) [ 0.225015] ACPI: Added _OSI(Processor Device) [ 0.227011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232819] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239539] ACPI: Interpreter enabled [ 0.241062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243015] ACPI: Using IOAPIC for interrupt routing [ 0.245148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249353] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259869] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273477] acpiphp: Slot [2] registered [ 0.275128] acpiphp: Slot [5] registered [ 0.277129] acpiphp: Slot [6] registered [ 0.279128] acpiphp: Slot [7] registered [ 0.280079] acpiphp: Slot [8] registered [ 0.282141] acpiphp: Slot [9] registered [ 0.283203] acpiphp: Slot [10] registered [ 0.285145] acpiphp: Slot [3] registered [ 0.287131] acpiphp: Slot [4] registered [ 0.289116] acpiphp: Slot [11] registered [ 0.291107] acpiphp: Slot [12] registered [ 0.292100] acpiphp: Slot [13] registered [ 0.294105] acpiphp: Slot [14] registered [ 0.295105] acpiphp: Slot [15] registered [ 0.296131] acpiphp: Slot [16] registered [ 0.297116] acpiphp: Slot [17] registered [ 0.298099] acpiphp: Slot [18] registered [ 0.299073] acpiphp: Slot [19] registered [ 0.301103] acpiphp: Slot [20] registered [ 0.302102] acpiphp: Slot [21] registered [ 0.303125] acpiphp: Slot [22] registered [ 0.305109] acpiphp: Slot [23] registered [ 0.307101] acpiphp: Slot [24] registered [ 0.308146] acpiphp: Slot [25] registered [ 0.309102] acpiphp: Slot [26] registered [ 0.311116] acpiphp: Slot [27] registered [ 0.312142] acpiphp: Slot [28] registered [ 0.314133] acpiphp: Slot [29] registered [ 0.315093] acpiphp: Slot [30] registered [ 0.316124] acpiphp: Slot [31] registered [ 0.318071] PCI host bridge to bus 0000:00 [ 0.320025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338068] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342371] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370558] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.372736] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378902] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.410709] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444914] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.464016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489497] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.531025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.544088] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.556019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.588206] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.597015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.624016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.634644] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.692515] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696436] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.699405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.702369] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.705295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709154] iommu: Default domain type: Passthrough [ 0.710000] SCSI subsystem initialized [ 0.711168] ACPI: bus type USB registered [ 0.712164] usbcore: registered new interface driver usbfs [ 0.715101] usbcore: registered new interface driver hub [ 0.717111] usbcore: registered new device driver usb [ 0.719249] pps_core: LinuxPPS API ver. 1 registered [ 0.722012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.725080] PTP clock support registered [ 0.727316] EDAC MC: Ver: 3.0.0 [ 0.730125] PCI: Using ACPI for IRQ routing [ 0.732961] NetLabel: Initializing [ 0.734014] NetLabel: domain hash size = 128 [ 0.736012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739127] NetLabel: unlabeled traffic allowed by default [ 0.741127] vgaarb: loaded [ 0.743294] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.751364] clocksource: Switched to clocksource kvm-clock [ 0.859066] VFS: Disk quotas dquot_6.6.0 [ 0.861509] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.864096] *** VALIDATE ramfs *** [ 0.865254] *** VALIDATE hugetlbfs *** [ 0.867167] pnp: PnP ACPI init [ 0.869757] pnp: PnP ACPI: found 6 devices [ 0.890810] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.895167] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.897630] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900923] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904552] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.907212] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.910402] NET: Registered protocol family 2 [ 0.913134] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918248] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.922446] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928778] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.932532] TCP: Hash tables configured (established 65536 bind 65536) [ 0.935799] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.939347] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942777] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946213] NET: Registered protocol family 1 [ 0.949334] RPC: Registered named UNIX socket transport module. [ 0.951747] RPC: Registered udp transport module. [ 0.953518] RPC: Registered tcp transport module. [ 0.954828] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.957361] NET: Registered protocol family 44 [ 0.959289] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961874] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.964323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.967110] PCI: CLS 0 bytes, default 64 [ 0.969375] Unpacking initramfs... [ 2.379838] debug: unmapping init [mem 0xffff9931bcc54000-0xffff9931bffbffff] [ 2.384164] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.386563] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.389553] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.906295] Initialise system trusted keyrings [ 2.908497] Key type blacklist registered [ 2.910652] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.919969] zbud: loaded [ 2.923626] *** VALIDATE nfs *** [ 2.925019] *** VALIDATE nfs4 *** [ 2.926889] pstore: using deflate compression [ 2.930585] Platform Keyring initialized [ 3.056900] NET: Registered protocol family 38 [ 3.059558] Key type asymmetric registered [ 3.062016] Asymmetric key parser 'x509' registered [ 3.064765] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.069352] io scheduler mq-deadline registered [ 3.072117] io scheduler kyber registered [ 3.074843] io scheduler bfq registered [ 3.077155] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.080735] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.083934] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.086981] ACPI: Power Button [PWRF] [ 3.092513] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.099752] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.117171] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.126149] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.141562] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.168744] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.197990] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.205834] Non-volatile memory driver v1.3 [ 3.207831] Linux agpgart interface v0.103 [ 3.248318] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.252017] vda: detected capacity change from 0 to 74825728 [ 3.279811] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.283143] vdb: detected capacity change from 0 to 1073741824 [ 3.298123] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.311273] vdc: detected capacity change from 0 to 2621440000 [ 3.334402] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.337515] vdd: detected capacity change from 0 to 2621440000 [ 3.352203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.354579] vde: detected capacity change from 0 to 4294967296 [ 3.369038] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.371396] vdf: detected capacity change from 0 to 4294967296 [ 3.377245] libphy: Fixed MDIO Bus: probed [ 3.382800] usbcore: registered new interface driver usbserial_generic [ 3.384762] usbserial: USB Serial support registered for generic [ 3.388499] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.393574] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.395898] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.399362] mousedev: PS/2 mouse device common for all mice [ 3.403875] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.405488] rtc_cmos 00:05: RTC can wake from S4 [ 3.412410] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.413408] rtc_cmos 00:05: registered as rtc0 [ 3.421109] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.422412] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.426038] intel_pstate: CPU model not supported [ 3.435618] hid: raw HID events driver (C) Jiri Kosina [ 3.438261] usbcore: registered new interface driver usbhid [ 3.440841] usbhid: USB HID core driver [ 3.442422] drop_monitor: Initializing network drop monitor service [ 3.445438] Initializing XFRM netlink socket [ 3.447665] NET: Registered protocol family 10 [ 3.451351] Segment Routing with IPv6 [ 3.453261] NET: Registered protocol family 17 [ 3.455916] mpls_gso: MPLS GSO support [ 3.461104] RAS: Correctable Errors collector initialized. [ 3.463808] AVX version of gcm_enc/dec engaged. [ 3.465860] AES CTR mode by8 optimization enabled [ 3.549546] sched_clock: Marking stable (3549513715, 0)->(4460462937, -910949222) [ 3.552523] registered taskstats version 1 [ 3.554356] Loading compiled-in X.509 certificates [ 3.556233] zswap: loaded using pool lzo/zbud [ 3.584399] Key type big_key registered [ 3.600726] Key type encrypted registered [ 3.602599] ima: No TPM chip found, activating TPM-bypass! [ 3.605226] ima: Allocated hash algorithm: sha1 [ 3.607205] ima: No architecture policies found [ 3.609200] evm: Initialising EVM extended attributes: [ 3.610894] evm: security.selinux [ 3.612094] evm: security.ima [ 3.615380] evm: security.capability [ 3.616860] evm: HMAC attrs: 0x1 [ 3.620126] rtc_cmos 00:05: setting system clock to 2026-08-24 10:50:33 UTC (1787568633) [ 3.627157] debug: unmapping init [mem 0xffffffff87203000-0xffffffff873fffff] [ 3.631040] debug: unmapping init [mem 0xffffffff85f82000-0xffffffff86258fff] [ 3.639120] Write protecting the kernel read-only data: 28672k [ 3.643212] debug: unmapping init [mem 0xffffffff84603000-0xffffffff847fffff] [ 3.646875] debug: unmapping init [mem 0xffffffff84f14000-0xffffffff84ffffff] [ 3.680483] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.690814] systemd[1]: Detected virtualization kvm. [ 3.693086] systemd[1]: Detected architecture x86-64. [ 3.695367] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.722598] systemd[1]: No hostname configured. [ 3.725085] systemd[1]: Set hostname to . [ 3.727631] random: systemd: uninitialized urandom read (16 bytes read) [ 3.730868] systemd[1]: Initializing machine ID from random generator. [ 3.781349] random: ln: uninitialized urandom read (6 bytes read) [ 3.877974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.881463] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.887484] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.892971] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.548990] device-mapper: uevent: version 1.0.3 [ 4.551544] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.335802] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.385847] virtio_net virtio0 ens2: renamed from eth0 [ 5.469801] scsi host0: ata_piix [ 5.484473] scsi host1: ata_piix [ 5.486363] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.489511] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.932375] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.123205] random: crng init done [ 10.124955] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.660366] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.841964] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.080406] SELinux: Disabled at runtime. [ 12.141560] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.151430] systemd[1]: Detected virtualization kvm. [ 12.153372] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.680811] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.684649] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.693132] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.697565] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.700802] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.711832] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.725440] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ 12.804222] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.173437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.521955] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.553551] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.636536] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.659525] EDAC sbridge: Ver: 1.1.2 [ 15.440104] Key type dns_resolver registered [ 15.782621] NFS: Registering the id_resolver key type [ 15.785323] Key type id_resolver registered [ 15.787245] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg441-server login: [ 25.663018] hrtimer: interrupt took 8430889 ns [ 74.409199] libcfs: loading out-of-tree module taints kernel. [ 74.475871] Key type ._llcrypt registered [ 74.484556] Key type .llcrypt registered [ 74.600638] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 94.160510] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 96.818488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.840157] alg: No test for adler32 (adler32-zlib) [ 98.735591] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 100.415898] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 102.248259] Key type lgssc registered [ 105.105535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.697380] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.137565] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 196.357313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.425437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.703308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.728736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.866786] Lustre: lustre-MDT0000: new disk, initializing [ 197.971589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.995031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.855450] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.046518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.159183] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.192371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.196605] Lustre: Skipped 1 previous similar message [ 217.304352] Lustre: lustre-MDT0001: new disk, initializing [ 217.414762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.444641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.467213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.893788] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.968610] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.605074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.940885] Lustre: lustre-OST0000: new disk, initializing [ 238.944630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.960902] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.053847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.950964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.967451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.035848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.696576] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.014208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.207767] Lustre: lustre-OST0001: new disk, initializing [ 262.213683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 262.221408] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.289601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.873149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.880066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 267.985930] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.015079] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.517289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 291.926122] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.419465] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing check_logdir /tmp/testlogs/ [ 306.296462] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing yml_node [ 311.017836] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 314.040130] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 317.093297] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 319.402133] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 06:55:47 EDT 2026 [ 341.543419] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.252815] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.422489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.438022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.446191] Lustre: Skipped 3 previous similar messages [ 365.548894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.570705] Lustre: Skipped 3 previous similar messages [ 370.659501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.669103] Lustre: Skipped 3 previous similar messages [ 374.240588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.495708] Lustre: server umount lustre-MDT0000 complete [ 378.573563] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569008 with bad export cookie 7364051985160722044 [ 378.580988] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.584216] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 380.897864] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.898263] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.898851] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.920037] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 380.954331] Lustre: Skipped 4 previous similar messages [ 383.905206] LustreError: 8891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.924455] LustreError: 8891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 385.007418] Lustre: server umount lustre-MDT0001 complete [ 389.290950] Lustre: server umount lustre-OST0000 complete [ 393.253740] Lustre: server umount lustre-OST0001 complete [ 400.172960] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 410.160648] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 449.996648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.429579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.479404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.684842] Lustre: lustre-MDT0000: new disk, initializing [ 450.930575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.979358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.699877] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 468.619219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.770694] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 468.851801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 468.855446] Lustre: Skipped 1 previous similar message [ 468.964380] Lustre: lustre-MDT0001: new disk, initializing [ 469.103441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 469.211568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 469.230247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.905173] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 485.753631] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 493.076468] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.144824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 501.500485] Lustre: lustre-OST0000: new disk, initializing [ 501.508367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 501.522674] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 501.646086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.911008] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 502.925154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 503.049081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 508.622973] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 519.198360] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 525.924456] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 526.236987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 532.241142] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 532.643073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 539.106899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.113471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.121459] LustreError: Skipped 1 previous similar message [ 539.138409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.158112] Lustre: Skipped 1 previous similar message [ 539.188318] Lustre: Skipped 4 previous similar messages [ 549.350632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.359370] Lustre: Skipped 2 previous similar messages [ 549.363618] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 549.608174] Lustre: server umount lustre-OST0000 complete [ 554.469610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.506242] Lustre: server umount lustre-MDT0000 complete [ 564.505180] LustreError: 15890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569194 with bad export cookie 7364051985160723584 [ 564.509914] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.523110] LustreError: 15890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 565.051493] Lustre: server umount lustre-MDT0001 complete [ 569.404392] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:59:57 (1787569197) [ 572.328506] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 574.550912] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:00:02 (1787569202) [ 581.748627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.189261] LustreError: 19467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.301996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.747727] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.908973] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 592.886044] LustreError: 19467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.381702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.643138] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 613.369173] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.227817] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 628.168786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.925699] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 644.386506] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 667.622545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.625423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.628429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 667.645036] Lustre: Skipped 2 previous similar messages [ 667.683995] Lustre: Skipped 6 previous similar messages [ 670.481832] Lustre: server umount lustre-OST0000 complete [ 674.786261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 677.857772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.868543] Lustre: Skipped 1 previous similar message [ 680.862751] Lustre: server umount lustre-MDT0000 complete [ 682.977546] LustreError: 21973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.005495] LustreError: 21973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 685.241573] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569315 with bad export cookie 7364051985160724466 [ 685.243618] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.255715] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 685.758432] Lustre: server umount lustre-MDT0001 complete [ 699.237423] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 703.217708] Key type lgssc unregistered [ 703.654937] LNet: 23043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.661662] LNetError: 23043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.678807] LNet: Removed LNI 192.168.204.141@tcp [ 704.839164] Key type .llcrypt unregistered [ 704.846452] Key type ._llcrypt unregistered [ 734.032327] Key type ._llcrypt registered [ 734.042238] Key type .llcrypt registered [ 734.196729] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 735.823738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.940986] alg: No test for adler32 (adler32-zlib) [ 737.320436] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 737.725393] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 739.480556] Key type lgssc registered [ 740.833477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.064834] loop: module loaded [ 751.173955] loop0: detected capacity change from 0 to 204800000 [ 751.442638] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.453652] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.739603] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.648828] loop0: detected capacity change from 0 to 204800000 [ 758.955774] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.975115] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.985763] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 765.575945] loop0: detected capacity change from 0 to 204800000 [ 771.426929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 771.477891] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.830564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 772.900986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 772.994887] Lustre: lustre-MDT0000: new disk, initializing [ 773.101657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.124188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 778.120096] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 787.199468] loop1: detected capacity change from 0 to 204800000 [ 793.900257] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.305775] Lustre: lustre-OST07c6: new disk, initializing [ 794.309588] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 794.313771] Lustre: Skipped 1 previous similar message [ 794.318445] Lustre: 26231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 794.408586] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 801.073081] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 803.465633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 803.489535] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 803.521423] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 805.864417] Lustre: 25218:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 805.878402] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 805.885660] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 805.890609] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 805.907329] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 805.915058] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 825.009417] Lustre: 26913:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 825.020994] Lustre: 26913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 825.030545] Lustre: 26913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 825.039098] Lustre: 26913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 825.049849] Lustre: 26913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 825.059374] Lustre: 26913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.615843] Lustre: 25219:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 830.629063] Lustre: 25219:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 830.644311] Lustre: 25219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 830.653813] Lustre: 25219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.670263] Lustre: 25219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.683933] Lustre: 25219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.696424] Lustre: 25219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 830.712873] Lustre: 25219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.726036] Lustre: 25219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 830.730835] Lustre: 25219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.739682] Lustre: 25219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.745950] Lustre: 25219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.038551] Lustre: 25220:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 834.052867] Lustre: 25220:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 834.061131] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 834.071167] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.079362] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 834.086095] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.101610] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 834.111405] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.125868] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 834.141362] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.149879] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 834.157803] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.104271] Lustre: 25220:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 839.121979] Lustre: 25220:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 839.127963] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 839.132204] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.139150] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 839.148913] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.166124] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 839.176244] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.187360] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 839.209222] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.218189] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 839.223165] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 850.810092] Lustre: 25218:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 850.823276] Lustre: 25218:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 850.835120] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 850.846390] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 850.854227] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 850.864358] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 850.876545] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 850.888634] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 850.900486] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 850.910444] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 850.922407] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 850.926513] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 900.586540] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.606379] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 905.702410] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 910.818557] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 913.376427] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 913.672671] Lustre: server umount lustre-OST07c6 complete [ 925.651552] Lustre: server umount lustre-MDT0000 complete [ 945.678375] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 948.272593] Key type lgssc unregistered [ 948.586745] LNet: 28793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.594126] LNetError: 28793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.618076] LNet: Removed LNI 192.168.204.141@tcp [ 949.603827] Key type .llcrypt unregistered [ 949.606411] Key type ._llcrypt unregistered [ 972.031897] Key type ._llcrypt registered [ 972.036081] Key type .llcrypt registered [ 972.176148] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 973.259923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.300230] alg: No test for adler32 (adler32-zlib) [ 974.272409] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 974.547915] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 976.216291] Key type lgssc registered [ 977.378610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.382425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.425755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.008792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.040832] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 997.351569] LustreError: 30067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.377980] LustreError: 30067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1002.470028] LustreError: 30068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.175266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.974800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.540914] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1022.262758] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.586975] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.245529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.743684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.993359] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1056.132338] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1057.142994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1077.754172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.775801] LustreError: 31890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.799638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.067928] Lustre: server umount lustre-OST0000 complete [ 1082.852308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.858956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.870132] Lustre: Skipped 2 previous similar messages [ 1082.896472] Lustre: Skipped 1 previous similar message [ 1087.976186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.984171] Lustre: Skipped 1 previous similar message [ 1089.127297] Lustre: server umount lustre-MDT0000 complete [ 1092.857382] LustreError: 30046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569722 with bad export cookie 10488341527476139130 [ 1092.860882] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.871522] LustreError: 30046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1093.336229] Lustre: server umount lustre-MDT0001 complete [ 1106.730776] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 1110.312221] Key type lgssc unregistered [ 1110.814859] LNet: 33810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.828239] LNetError: 33810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.851579] LNet: Removed LNI 192.168.204.141@tcp [ 1111.831380] Key type .llcrypt unregistered [ 1111.834626] Key type ._llcrypt unregistered [ 1133.644633] Key type ._llcrypt registered [ 1133.648519] Key type .llcrypt registered [ 1133.780581] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 1135.126596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1135.238072] alg: No test for adler32 (adler32-zlib) [ 1136.433800] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1136.785853] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 1138.536177] Key type lgssc registered [ 1139.629954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1147.727701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1147.749072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.166670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.760335] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1158.631204] LustreError: 35088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.660871] LustreError: 35088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1162.463122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.729609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.363751] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1178.119218] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1184.555175] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.099941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.585516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.682955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1199.757947] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1208.156727] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1223.656414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1223.659889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.668673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.709340] Lustre: Skipped 1 previous similar message [ 1226.992180] Lustre: server umount lustre-OST0000 complete [ 1233.891982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.921131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.930478] Lustre: Skipped 2 previous similar messages [ 1237.792329] Lustre: server umount lustre-MDT0000 complete [ 1239.010113] LustreError: 35082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.051275] LustreError: 35082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1242.013719] LustreError: 35069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787569871 with bad export cookie 15354025112087977908 [ 1242.045310] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.372335] Lustre: server umount lustre-MDT0001 complete [ 1255.095620] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 1258.454444] Key type lgssc unregistered [ 1258.789119] LNet: 38638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.810678] LNetError: 38638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.839363] LNet: Removed LNI 192.168.204.141@tcp [ 1259.899763] Key type .llcrypt unregistered [ 1259.902369] Key type ._llcrypt unregistered [ 1289.761796] Key type ._llcrypt registered [ 1289.763494] Key type .llcrypt registered [ 1289.869984] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 1290.985636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.027649] alg: No test for adler32 (adler32-zlib) [ 1292.148546] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1292.344576] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 1294.072584] Key type lgssc registered [ 1295.207781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.322973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.342165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.859084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.472375] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1315.304835] LustreError: 40058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.329643] LustreError: 40058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1319.394410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.763416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1324.147833] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1332.945903] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1340.310690] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1347.820106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.298529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.393586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1355.534346] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1363.944893] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1371.106903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1371.111691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.121170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.204897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.216047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.231190] Lustre: Skipped 1 previous similar message [ 1375.242994] Lustre: Skipped 3 previous similar messages [ 1376.628844] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) [ 1377.299195] Lustre: server umount lustre-MDT0000 complete [ 1380.326532] LustreError: 41901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.338519] LustreError: 41901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1381.303878] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570011 with bad export cookie 17852999841562581842 [ 1381.315652] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1381.320538] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.746284] LustreError: 40054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.770199] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.41@tcp (stopping) [ 1385.441880] LustreError: 41901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.442238] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.470197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1386.477527] Lustre: Skipped 1 previous similar message [ 1387.769551] Lustre: server umount lustre-MDT0001 complete [ 1409.459207] Lustre: server umount lustre-OST0000 complete [ 1427.806526] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 1430.715775] Key type lgssc unregistered [ 1431.065026] LNet: 43922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.074731] LNetError: 43922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.099208] LNet: Removed LNI 192.168.204.141@tcp [ 1432.039801] Key type .llcrypt unregistered [ 1432.041325] Key type ._llcrypt unregistered [ 1456.646073] Key type ._llcrypt registered [ 1456.647527] Key type .llcrypt registered [ 1456.771731] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 1457.606117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.661965] alg: No test for adler32 (adler32-zlib) [ 1458.986780] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1459.188899] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 1460.888211] Key type lgssc registered [ 1462.239970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.202598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.221655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.024109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.096557] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1482.219657] LustreError: 45339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.241845] LustreError: 45339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1487.349675] LustreError: 45338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.992122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.395703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.231346] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1501.393847] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1507.784903] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1515.815993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.296931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.309056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1523.860523] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1532.637690] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1539.047550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.052135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.066211] Lustre: Skipped 1 previous similar message [ 1539.084776] Lustre: Skipped 1 previous similar message [ 1544.180520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.195515] Lustre: Skipped 2 previous similar messages [ 1549.301836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1549.307146] Lustre: Skipped 2 previous similar messages [ 1552.352142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1552.526456] Lustre: server umount lustre-OST0000 complete [ 1574.882334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.887877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.898062] Lustre: Skipped 1 previous similar message [ 1574.909827] Lustre: Skipped 2 previous similar messages [ 1577.382178] Lustre: server umount lustre-MDT0000 complete [ 1580.004828] LustreError: 45334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.015820] LustreError: 45334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1582.493504] LustreError: 45319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570212 with bad export cookie 16196818733591706733 [ 1582.505964] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.509973] LustreError: 45319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1589.153627] Lustre: server umount lustre-MDT0001 complete [ 1601.820685] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 1605.100521] Key type lgssc unregistered [ 1605.494812] LNet: 49056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.511684] LNetError: 49056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.553218] LNet: Removed LNI 192.168.204.141@tcp [ 1606.826922] Key type .llcrypt unregistered [ 1606.830322] Key type ._llcrypt unregistered [ 1633.087701] Key type ._llcrypt registered [ 1633.091349] Key type .llcrypt registered [ 1633.219858] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 1634.166601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1634.204186] alg: No test for adler32 (adler32-zlib) [ 1635.201900] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1635.403619] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 1637.088215] Key type lgssc registered [ 1638.464677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.483780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1646.505870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.025463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.498832] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1657.321433] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.345328] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1660.380307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.737732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1664.763731] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1671.938656] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1677.658762] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.822296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.262988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.462787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1693.443645] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1703.649407] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1710.526538] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1714.305841] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1726.523624] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 7h19m14s [ 1728.177789] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1732.522965] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) [ 1732.580943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.592426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.617957] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) [ 1733.642284] Lustre: Skipped 3 previous similar messages [ 1737.955860] Lustre: server umount lustre-MDT0000 complete [ 1738.671835] LustreError: 51761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.747814] LustreError: 50473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.794616] LustreError: 50473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1742.685478] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1744.753427] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.914165] LustreError: 51761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.930786] LustreError: 51761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1751.626071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.838592] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.049249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.135390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1753.093643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1758.541568] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1763.083960] Lustre: DEBUG MARKER: Wait for df (36810) ... [ 1764.952932] Lustre: DEBUG MARKER: done [ 1772.513544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.523916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.530946] Lustre: Skipped 2 previous similar messages [ 1772.544941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1772.553680] Lustre: Skipped 3 previous similar messages [ 1773.543382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.547345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.590294] Lustre: server umount lustre-OST0000 complete [ 1783.783153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.792204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1783.814190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.839068] Lustre: Skipped 3 previous similar messages [ 1786.155820] Lustre: server umount lustre-MDT0000 complete [ 1790.370206] LustreError: 50452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570420 with bad export cookie 12006331512186181827 [ 1790.385451] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.386286] LustreError: 50452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1790.798644] Lustre: server umount lustre-MDT0001 complete [ 1805.359345] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 1808.525564] Key type lgssc unregistered [ 1808.907601] LNet: 55401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.917326] LNetError: 55401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.935832] LNet: Removed LNI 192.168.204.141@tcp [ 1809.813307] Key type .llcrypt unregistered [ 1809.819047] Key type ._llcrypt unregistered [ 1857.073555] Key type ._llcrypt registered [ 1857.077393] Key type .llcrypt registered [ 1857.274741] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 1858.834651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.929794] alg: No test for adler32 (adler32-zlib) [ 1859.917394] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 1860.139392] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 1861.816224] Key type lgssc registered [ 1863.242032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.030667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.058410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.439367] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1874.466962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1874.651926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.337684] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1888.774373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.899139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1888.960892] Lustre: 57612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1889.314069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.309403] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1902.281820] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1908.487574] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1916.158884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.396775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1916.706315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.552330] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 1924.264618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1933.536364] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1942.296558] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 7h22m49s [ 1944.963477] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1954.209775] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1956.894598] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1956.946451] LustreError: 57619:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1977.267076] Lustre: lustre-MDT0000: Client ef37510d-f02c-4cf0-b576-486063540754 (at 192.168.204.41@tcp) reconnecting [ 1996.992766] LustreError: 57619:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1999.704319] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2002.284408] Lustre: DEBUG MARKER: done [ 2011.105660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.119277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.153536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.617391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.631123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.959366] Lustre: server umount lustre-OST0000 complete [ 2021.858361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.881535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.892694] Lustre: Skipped 2 previous similar messages [ 2022.947254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.963756] Lustre: Skipped 1 previous similar message [ 2029.031208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.036345] Lustre: Skipped 1 previous similar message [ 2033.123248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.149438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2034.322509] Lustre: server umount lustre-MDT0000 complete [ 2038.298242] LustreError: 57603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570668 with bad export cookie 5115314045619592415 [ 2038.300935] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.322184] LustreError: 57603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2038.706461] Lustre: server umount lustre-MDT0001 complete [ 2050.423629] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 2053.398758] Key type lgssc unregistered [ 2053.719851] LNet: 61948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.726838] LNetError: 61948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.756036] LNet: Removed LNI 192.168.204.141@tcp [ 2054.660280] Key type .llcrypt unregistered [ 2054.665209] Key type ._llcrypt unregistered [ 2105.270289] Key type ._llcrypt registered [ 2105.272922] Key type .llcrypt registered [ 2105.373107] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 2106.676257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2106.851259] alg: No test for adler32 (adler32-zlib) [ 2108.014643] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2108.315178] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 2110.000213] Key type lgssc registered [ 2111.145830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.667223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.703498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.998688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2116.034362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2116.186064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.149701] Lustre: Failing over lustre-MDT0000 [ 2117.423439] Lustre: server umount lustre-MDT0000 complete [ 2124.427158] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:25:52 (1787570752) [ 2131.297084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.685687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.527094] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2146.953614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.098615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2147.157518] Lustre: 64748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2147.508593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.292559] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2160.945553] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.641750] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.302532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.440938] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2176.614248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.724266] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2185.929639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2191.325381] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2197.858351] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2206.690839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.695391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.708377] Lustre: Skipped 1 previous similar message [ 2206.722176] Lustre: Skipped 2 previous similar messages [ 2211.937122] Lustre: server umount lustre-MDT0000 complete [ 2215.672119] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570845 with bad export cookie 1377984227646084142 [ 2215.678980] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.692505] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2216.077089] Lustre: server umount lustre-MDT0001 complete [ 2220.335071] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2225.800132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2233.056210] Lustre: 63673:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570846/real 1787570846] req@00000000ce4be23f x1874403774210432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2233.079440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.094763] Lustre: Skipped 1 previous similar message [ 2234.880847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.385405] Lustre: 63671:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570850/real 1787570850] req@0000000043a4e716 x1874403774210560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2237.408150] Lustre: 63670:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570851/real 1787570851] req@0000000019254daf x1874403774210688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2240.290178] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x131f955aaaef9d11 [ 2240.303085] Lustre: MGC192.168.204.141@tcp: Connection restored to 0@lo (at 0@lo) [ 2240.787374] LustreError: 66597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.808280] LustreError: 66597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2240.893265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.009037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2241.888195] Lustre: 63672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570855/real 1787570855] req@0000000020fae0f0 x1874403774210944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2241.901238] LustreError: 66598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.915858] LustreError: 66597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.926846] LustreError: 66597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2245.646557] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2248.035187] Lustre: 63672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570861/real 1787570861] req@00000000f821539c x1874403774211072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.161933] LustreError: 66598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.188884] LustreError: 66598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2253.285817] LustreError: 68518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.311482] LustreError: 68518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2255.907644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.380682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.496120] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2261.586492] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2270.112804] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2276.543444] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2287.076850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.093590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2287.094844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.109190] Lustre: Skipped 2 previous similar messages [ 2292.195656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.205224] Lustre: Skipped 2 previous similar messages [ 2297.317347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.325233] Lustre: Skipped 2 previous similar messages [ 2298.336677] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2298.679692] Lustre: server umount lustre-MDT0000 complete [ 2302.440092] LustreError: 68517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.477043] LustreError: 68517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2302.756589] LustreError: 64742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787570932 with bad export cookie 1377984227646086417 [ 2302.759072] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.770355] LustreError: 64742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.305785] Lustre: server umount lustre-MDT0001 complete [ 2309.795948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2320.086141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.936879] Lustre: 63672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787570937/real 1787570937] req@0000000073ab4bcc x1874403774240384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787570953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2323.971465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.990520] Lustre: Skipped 2 previous similar messages [ 2328.589068] LustreError: 69253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.742343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.860118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2332.855823] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2332.868882] Lustre: Skipped 1 previous similar message [ 2333.077700] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2342.922772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.969537] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2349.050229] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2358.987114] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2366.879104] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2374.303782] Lustre: server umount lustre-MDT0000 complete [ 2374.626043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.632415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.645336] Lustre: Skipped 1 previous similar message [ 2374.645941] LustreError: 71204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.645947] LustreError: 71204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2378.645977] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571008 with bad export cookie 1377984227646087796 [ 2378.650116] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.665543] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2379.131593] Lustre: server umount lustre-MDT0001 complete [ 2386.011763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2389.671704] Lustre: DEBUG MARKER: files compared the same [ 2402.223465] Lustre: server umount lustre-OST0000 complete [ 2421.365726] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 2424.614381] Key type lgssc unregistered [ 2424.978086] LNet: 74482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.990986] LNetError: 74482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.008203] LNet: Removed LNI 192.168.204.141@tcp [ 2425.990973] Key type .llcrypt unregistered [ 2425.999166] Key type ._llcrypt unregistered [ 2454.209773] Key type ._llcrypt registered [ 2454.212189] Key type .llcrypt registered [ 2454.419549] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 2456.415074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2456.572290] alg: No test for adler32 (adler32-zlib) [ 2457.650610] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2457.861851] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 2459.544545] Key type lgssc registered [ 2461.128382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2470.960220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2470.978535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.635306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.719297] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug +malloc all [ 2482.145934] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.171854] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2487.265417] LustreError: 75918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.811115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.276828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.824123] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug +malloc all [ 2502.013639] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2509.380102] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2517.226104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2517.684201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.724742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2525.691175] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug +malloc all [ 2534.324713] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2544.610619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2544.612956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.625618] LustreError: Skipped 1 previous similar message [ 2544.626955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2544.651564] Lustre: Skipped 1 previous similar message [ 2549.686252] Lustre: server umount lustre-OST0000 complete [ 2554.849257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.856163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.858057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.858068] Lustre: Skipped 1 previous similar message [ 2554.873575] Lustre: Skipped 1 previous similar message [ 2559.974191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.979724] Lustre: Skipped 1 previous similar message [ 2565.089284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.096706] Lustre: Skipped 1 previous similar message [ 2568.160402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2568.371871] Lustre: server umount lustre-MDT0000 complete [ 2570.218319] LustreError: 77218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.257845] LustreError: 77218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2572.602046] LustreError: 75896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571202 with bad export cookie 274552444239633029 [ 2572.607198] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.611161] LustreError: 75896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2572.929771] Lustre: server umount lustre-MDT0001 complete [ 2585.542693] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 2588.731687] Key type lgssc unregistered [ 2589.094598] LNet: 79456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.100602] LNetError: 79456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.117627] LNet: Removed LNI 192.168.204.141@tcp [ 2590.061817] Key type .llcrypt unregistered [ 2590.065046] Key type ._llcrypt unregistered [ 2615.747729] Key type ._llcrypt registered [ 2615.749986] Key type .llcrypt registered [ 2615.837597] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 2616.768351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.866046] alg: No test for adler32 (adler32-zlib) [ 2617.995981] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2618.199719] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 2619.923661] Key type lgssc registered [ 2621.039765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2629.219713] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2629.253869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.834177] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2649.955315] LustreError: 80849:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.141@tcp: failed processing log, type 4: rc = -110 [ 2676.640235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.251333] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2690.465589] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2697.701421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.834376] Lustre: *** cfs_fail_loc=706, val=0*** [ 2698.231304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.324347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2702.507216] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2703.334416] LustreError: 80875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.344866] LustreError: 80875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2708.450765] LustreError: 80874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.505185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.840699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.517820] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2723.019352] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2728.347945] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.495162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2731.503713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.532707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.522055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.525416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.284596] Lustre: server umount lustre-OST0000 complete [ 2741.729930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2741.735461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.748247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.752912] Lustre: Skipped 1 previous similar message [ 2745.642398] Lustre: server umount lustre-MDT0000 complete [ 2747.874992] LustreError: 81938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.910281] LustreError: 81938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2749.152751] LustreError: 80855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571379 with bad export cookie 18346246181972965707 [ 2749.160932] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.169594] LustreError: 80855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2749.409276] Lustre: server umount lustre-MDT0001 complete [ 2759.558699] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 2762.131772] Key type lgssc unregistered [ 2762.493631] LNet: 84466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.506193] LNetError: 84466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.570477] LNet: Removed LNI 192.168.204.141@tcp [ 2764.594924] Key type .llcrypt unregistered [ 2764.599214] Key type ._llcrypt unregistered [ 2789.419473] Key type ._llcrypt registered [ 2789.424643] Key type .llcrypt registered [ 2789.548486] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 2790.799664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2790.884836] alg: No test for adler32 (adler32-zlib) [ 2792.042179] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 2792.359747] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 2794.065085] Key type lgssc registered [ 2795.439910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2804.269136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2804.301286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.554658] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2818.967688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.460917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.574649] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2824.673633] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.692635] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2829.794215] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.830944] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2833.021765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.384227] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.411025] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2833.411260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.502648] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.537555] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2840.714931] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2843.619338] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.633597] LustreError: 86571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2849.966692] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.980760] Lustre: MGS: Client 53bbbdaf-025a-439e-be65-9365ff5bd438 (at 0@lo) reconnecting [ 2850.010370] LustreError: 86585:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2850.362904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.505194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2855.815464] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 2870.756198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2870.779625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.789197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2870.798749] Lustre: Skipped 1 previous similar message [ 2870.806180] Lustre: Skipped 1 previous similar message [ 2875.533818] Lustre: server umount lustre-OST0000 complete [ 2880.993101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2881.007133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.020697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.065545] Lustre: Skipped 1 previous similar message [ 2886.115582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.126982] Lustre: Skipped 2 previous similar messages [ 2886.511878] Lustre: server umount lustre-MDT0000 complete [ 2890.211635] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.240599] LustreError: 86574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2892.009737] Lustre: server umount lustre-MDT0001 complete [ 2897.549354] Lustre: server umount MGS complete [ 2915.128924] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:39:03 (1787571543) [ 2951.791215] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 2962.941328] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3006.577929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.569673] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3024.183028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.900797] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3024.997392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3025.263438] Lustre: lustre-MDT0001: new disk, initializing [ 3025.419447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.483028] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3025.499632] Lustre: 93747:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3025.522693] LustreError: 93747:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3031.028541] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3043.662413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.130153] Lustre: lustre-OST0000: new disk, initializing [ 3044.140454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3044.149376] Lustre: 94512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3044.312298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.935575] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3052.641126] Lustre: 94830:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000aab42ce2 x1874404491436928/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787571698 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3064.105553] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3072.026905] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.044786] Lustre: MGS: Client 1c113472-b6b0-402b-87fe-b88351400f15 (at 0@lo) reconnecting [ 3072.105364] Lustre: 93012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3072.127085] LustreError: 93733:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3072.173235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3072.246366] Lustre: lustre-MDT0000: new disk, initializing [ 3072.352674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.376061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3072.392368] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3073.441843] Lustre: 95583:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001a9bf5a0 x1874404491444352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787571719 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3077.451788] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3078.688852] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3082.555893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3082.573324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3082.777954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3108.836254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.848648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.862981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.868201] Lustre: Skipped 1 previous similar message [ 3114.645783] Lustre: server umount lustre-OST0000 complete [ 3122.660809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3122.662416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.671690] LustreError: Skipped 1 previous similar message [ 3122.672783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.672791] Lustre: Skipped 2 previous similar messages [ 3122.704263] Lustre: Skipped 2 previous similar messages [ 3124.960970] Lustre: server umount lustre-MDT0000 complete [ 3127.778319] LustreError: 95834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.796491] LustreError: 95834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3134.928868] Lustre: server umount lustre-MDT0001 complete [ 3139.312645] Lustre: server umount MGS complete [ 3152.079661] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:43:00 (1787571780) [ 3172.800207] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 3176.110646] Key type lgssc unregistered [ 3176.383893] LNet: 98086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3176.390703] LNetError: 98086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3176.410730] LNet: Removed LNI 192.168.204.141@tcp [ 3177.327265] Key type .llcrypt unregistered [ 3177.330330] Key type ._llcrypt unregistered [ 3194.928469] Key type ._llcrypt registered [ 3194.931671] Key type .llcrypt registered [ 3195.062664] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3196.307420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.458822] alg: No test for adler32 (adler32-zlib) [ 3197.502522] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 3197.805218] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 3199.496167] Key type lgssc registered [ 3200.817363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.838321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3209.862092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.452274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.997138] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3221.988499] LustreError: 99308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.000809] LustreError: 99308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3227.107659] LustreError: 99309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.302578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.714839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.553043] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3240.782245] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3248.402867] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3257.506647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3258.106612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.574442] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3274.007660] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3281.658636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3293.153732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.162860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.179373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.251499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.255046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.256840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.264034] Lustre: Skipped 1 previous similar message [ 3299.225802] Lustre: server umount lustre-OST0000 complete [ 3304.417475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3304.423349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.432374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.490801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.491521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.522799] Lustre: Skipped 1 previous similar message [ 3309.639572] Lustre: server umount lustre-MDT0000 complete [ 3312.610790] LustreError: 99305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.632252] LustreError: 99305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3313.530088] LustreError: 99288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787571943 with bad export cookie 16823570313109514022 [ 3313.535735] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.543840] LustreError: 99288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3314.005296] Lustre: server umount lustre-MDT0001 complete [ 3325.774562] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 3329.285369] Key type lgssc unregistered [ 3329.543193] LNet: 102876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.554664] LNetError: 102876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3329.581996] LNet: Removed LNI 192.168.204.141@tcp [ 3330.753812] Key type .llcrypt unregistered [ 3330.757397] Key type ._llcrypt unregistered [ 3348.788661] Key type ._llcrypt registered [ 3348.791965] Key type .llcrypt registered [ 3348.940663] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3350.173455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3350.208921] alg: No test for adler32 (adler32-zlib) [ 3351.356850] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 3351.626899] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 3353.329193] Key type lgssc registered [ 3354.514526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3362.536525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3362.554343] LustreError: 103953:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3362.582363] LustreError: 103962:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3362.588769] LustreError: 103953:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3362.606581] LustreError: 103953:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3362.621366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.621848] LustreError: 103953:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3364.373761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.566885] LustreError: 104000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.605026] LustreError: 104000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3375.003728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.459672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.772882] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3386.586689] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing lsmod [ 3390.341928] LustreError: 105168:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3390.351388] LustreError: 105169:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3390.357983] LustreError: 105168:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3390.367121] LustreError: 105168:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3390.378470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.380294] LustreError: 105168:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3390.879986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.932173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3396.068910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3396.083463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.094582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.107902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.114299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.122556] Lustre: Skipped 1 previous similar message [ 3397.163142] Lustre: Skipped 1 previous similar message [ 3400.228738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.234721] Lustre: Skipped 2 previous similar messages [ 3401.623601] Lustre: server umount lustre-MDT0000 complete [ 3405.286757] LustreError: 103994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.317631] LustreError: 103994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3407.951762] LustreError: 103977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572037 with bad export cookie 8134665547903855398 [ 3407.963509] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.974786] LustreError: 103977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3408.440292] LustreError: 104432:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000325cbf14 x1874405077572096/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3408.616023] Lustre: server umount lustre-MDT0001 complete [ 3423.339454] Lustre: server umount lustre-OST0000 complete [ 3432.508442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.320986] LustreError: 106233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.389242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.565681] LustreError: 106234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.046535] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3443.688017] LustreError: 106233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.650300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.271428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.641255] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3468.603828] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.812746] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.564342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.305725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.467691] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3503.372119] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3503.743184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3513.326354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.352853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.380967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.389148] Lustre: Skipped 1 previous similar message [ 3514.361147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.305688] Lustre: server umount lustre-OST0000 complete [ 3524.580609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.590586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.612745] Lustre: Skipped 2 previous similar messages [ 3524.632124] Lustre: Skipped 3 previous similar messages [ 3526.756643] Lustre: server umount lustre-MDT0000 complete [ 3529.699514] LustreError: 106233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.733646] LustreError: 106233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3530.753801] LustreError: 106217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572160 with bad export cookie 8134665547903856329 [ 3530.765957] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.767827] LustreError: 106217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3531.186993] Lustre: server umount lustre-MDT0001 complete [ 3543.871670] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 3547.028580] Key type lgssc unregistered [ 3547.347828] LNet: 109774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3547.361391] LNetError: 109774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3547.376943] LNet: Removed LNI 192.168.204.141@tcp [ 3548.308647] Key type .llcrypt unregistered [ 3548.311442] Key type ._llcrypt unregistered [ 3577.314149] Key type ._llcrypt registered [ 3577.316889] Key type .llcrypt registered [ 3577.486657] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3578.899945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3579.099720] alg: No test for adler32 (adler32-zlib) [ 3580.308348] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 3580.699746] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 3582.496406] Key type lgssc registered [ 3583.688383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3593.139652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3593.161167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.630056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.091762] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3604.968981] LustreError: 111192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.985296] LustreError: 111192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3608.173939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.645294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3613.020625] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3620.374488] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3626.822414] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3633.447892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.778311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3639.786870] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3639.915053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3647.536559] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3652.458776] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3656.595217] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3665.896614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.904263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3665.915403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3665.922189] Lustre: Skipped 1 previous similar message [ 3665.944719] LustreError: Skipped 1 previous similar message [ 3667.444994] Lustre: server umount lustre-OST0000 complete [ 3675.109150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3675.122786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.146198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.154130] Lustre: Skipped 1 previous similar message [ 3676.140500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.163247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.172560] Lustre: Skipped 1 previous similar message [ 3677.870624] Lustre: server umount lustre-MDT0000 complete [ 3681.251372] LustreError: 112206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.263405] LustreError: 112206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3682.338790] LustreError: 111174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572312 with bad export cookie 13819347661369829323 [ 3682.349723] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.358471] LustreError: 111174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3682.725356] Lustre: server umount lustre-MDT0001 complete [ 3695.379572] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 3698.204925] Key type lgssc unregistered [ 3698.537636] LNet: 114834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3698.553427] LNetError: 114834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3698.575844] LNet: Removed LNI 192.168.204.141@tcp [ 3699.433356] Key type .llcrypt unregistered [ 3699.435498] Key type ._llcrypt unregistered [ 3718.212576] Key type ._llcrypt registered [ 3718.215674] Key type .llcrypt registered [ 3718.352491] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3720.058640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.083538] alg: No test for adler32 (adler32-zlib) [ 3721.361294] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 3721.745766] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 3723.472201] Key type lgssc registered [ 3725.011024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3734.503654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3734.528900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.984940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.998935] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3745.255599] LustreError: 116055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.291717] LustreError: 116055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3750.371165] LustreError: 116056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.391619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.821596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3757.301855] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3767.149801] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.701940] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.861583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.295283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.335457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3789.628769] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3799.712651] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3812.331055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.333291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.407057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.424062] Lustre: Skipped 1 previous similar message [ 3814.067824] Lustre: server umount lustre-OST0000 complete [ 3822.562185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.570510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3822.592663] Lustre: Skipped 2 previous similar messages [ 3826.443333] Lustre: server umount lustre-MDT0000 complete [ 3827.685076] LustreError: 116504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.713190] LustreError: 116504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3831.830713] LustreError: 116038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572461 with bad export cookie 1031248961409769788 [ 3831.833396] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3831.846580] LustreError: 116038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3832.197115] Lustre: server umount lustre-MDT0001 complete [ 3844.991323] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 3848.053025] Key type lgssc unregistered [ 3848.355932] LNet: 119595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3848.379193] LNetError: 119595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3848.422681] LNet: Removed LNI 192.168.204.141@tcp [ 3849.514522] Key type .llcrypt unregistered [ 3849.518523] Key type ._llcrypt unregistered [ 3877.273732] Key type ._llcrypt registered [ 3877.279405] Key type .llcrypt registered [ 3877.455111] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 3878.448453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3878.614137] alg: No test for adler32 (adler32-zlib) [ 3879.693132] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 3879.867029] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 3881.536163] Key type lgssc registered [ 3882.692912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.085454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.117886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.543208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.234453] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3902.956282] LustreError: 121011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.994286] LustreError: 121011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3906.879699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.317265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.542990] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3921.833371] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3928.103575] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3935.804767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.374316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.397243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3942.839156] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 3950.988692] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3961.113736] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3965.431809] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3965.438686] Lustre: Skipped 2 previous similar messages [ 3970.663775] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3970.700791] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3970.737507] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3970.743584] Lustre: Skipped 2 previous similar messages [ 3979.887746] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3979.898548] Lustre: Skipped 1 previous similar message [ 3984.453767] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.41@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3984.461764] Lustre: Skipped 2 previous similar messages [ 3989.611639] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.41@tcp 192.168.0.[2,10]@tcp [ 3998.193968] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.41/24@tcp in log lustre-MDT0000 [ 3998.199025] Lustre: Skipped 2 previous similar messages [ 3999.479529] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.41/24@tcp [ 3999.486459] Lustre: Skipped 1 previous similar message [ 4009.448201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.452324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.453281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.472467] Lustre: Skipped 1 previous similar message [ 4014.566869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4014.577336] Lustre: Skipped 2 previous similar messages [ 4019.685376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.697253] Lustre: Skipped 1 previous similar message [ 4022.240405] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4022.551089] Lustre: server umount lustre-OST0000 complete [ 4029.928380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.938646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.944536] Lustre: Skipped 1 previous similar message [ 4029.961159] Lustre: Skipped 1 previous similar message [ 4033.068325] Lustre: server umount lustre-MDT0000 complete [ 4035.047418] LustreError: 122000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.069627] LustreError: 122000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4037.087194] LustreError: 120991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572666 with bad export cookie 495839394340213077 [ 4037.093922] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4037.098647] LustreError: 120991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4037.663720] Lustre: server umount lustre-MDT0001 complete [ 4050.027708] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 4052.953965] Key type lgssc unregistered [ 4053.385789] LNet: 125960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4053.392607] LNetError: 125960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4053.428249] LNet: Removed LNI 192.168.204.141@tcp [ 4054.361191] Key type .llcrypt unregistered [ 4054.365812] Key type ._llcrypt unregistered [ 4085.131390] Key type ._llcrypt registered [ 4085.133315] Key type .llcrypt registered [ 4085.214813] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 4086.118384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4086.195337] alg: No test for adler32 (adler32-zlib) [ 4087.397368] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 4087.731418] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 4089.408348] Key type lgssc registered [ 4090.801929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4099.303117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4099.347986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.888557] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4100.919661] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.41/24@tcp [ 4100.947262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.848615] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4110.313039] LustreError: 127527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.333261] LustreError: 127527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4115.438360] LustreError: 127526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.005283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.438546] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4116.482721] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.41/24@tcp [ 4116.541959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4121.616748] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4132.174863] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.117746] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4148.911754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.357637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.890472] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4166.013473] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4170.915921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4182.499878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.517208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.550822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.596857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.597573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.929389] Lustre: server umount lustre-OST0000 complete [ 4196.837031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.853557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.861328] Lustre: Skipped 1 previous similar message [ 4199.828702] Lustre: server umount lustre-MDT0000 complete [ 4201.975386] LustreError: 127521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.998353] LustreError: 127521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4204.751943] LustreError: 127507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787572834 with bad export cookie 1536119747772190882 [ 4204.780890] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.213760] Lustre: server umount lustre-MDT0001 complete [ 4218.770229] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 4222.340523] Key type lgssc unregistered [ 4222.802616] LNet: 131120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4222.814314] LNetError: 131120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4223.849887] LNet: Removed LNI 192.168.204.141@tcp [ 4224.845757] Key type .llcrypt unregistered [ 4224.847846] Key type ._llcrypt unregistered [ 4256.984429] Key type ._llcrypt registered [ 4256.988300] Key type .llcrypt registered [ 4257.084612] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 4273.701814] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 4275.190055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4275.211886] alg: No test for adler32 (adler32-zlib) [ 4276.314611] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 4276.577141] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 4278.296227] Key type lgssc registered [ 4279.699471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4319.871396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4319.945529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.216847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4321.250263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4321.360403] Lustre: lustre-MDT0000: new disk, initializing [ 4321.484431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.504888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4326.438087] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4337.471091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.632892] Lustre: 134535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4337.695255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4337.699699] Lustre: Skipped 1 previous similar message [ 4337.782323] Lustre: lustre-MDT0001: new disk, initializing [ 4337.859905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4337.899775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4337.914050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4343.297628] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4353.939700] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.945357] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4367.940024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.260779] Lustre: lustre-OST0000: new disk, initializing [ 4368.269045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4368.279410] Lustre: 136501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4368.421146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4375.441649] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4377.695283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4377.709251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4377.856374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4386.144756] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.226905] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4391.536821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4398.276930] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4398.595589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4403.685753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4403.701170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.728058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4403.736074] Lustre: Skipped 1 previous similar message [ 4407.161145] Lustre: server umount lustre-OST0000 complete [ 4413.921369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.949233] Lustre: Skipped 1 previous similar message [ 4413.970475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.978594] Lustre: Skipped 1 previous similar message [ 4420.238407] Lustre: server umount lustre-MDT0000 complete [ 4424.793696] LustreError: 134525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573054 with bad export cookie 10352330219657645449 [ 4424.796818] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4424.813776] LustreError: 134525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4425.206636] Lustre: server umount lustre-MDT0001 complete [ 4434.681186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.299093] LustreError: 138725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4435.318307] LustreError: 138725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4435.437380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.190784] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4440.558246] LustreError: 138726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.669077] LustreError: 138725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.754789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.792712] LustreError: 138726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.231872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.369866] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4464.659124] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4470.698319] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4478.191585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4478.598619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.602948] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4493.651681] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4498.251118] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4498.434499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4503.623208] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4503.820354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4513.836262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.031371] Lustre: lustre-OST0001: new disk, initializing [ 4514.038130] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4514.045364] Lustre: 142096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4514.151033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4521.660053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4521.675621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4521.823393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4522.190588] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4532.312827] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4532.686421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4538.126670] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4538.533971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4545.617514] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a04e037c800.ost_server_uuid 50 [ 4547.470971] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a04e037c800.ost_server_uuid in IDLE state after 0 sec [ 4562.931355] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.936209] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4562.937026] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4562.937031] LustreError: Skipped 1 previous similar message [ 4562.943342] Lustre: Skipped 2 previous similar messages [ 4562.971902] Lustre: Skipped 1 previous similar message [ 4566.685973] Lustre: server umount lustre-OST0001 complete [ 4568.064638] LustreError: 140549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.088416] LustreError: 140549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4573.167371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4573.172080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.184803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.202637] Lustre: Skipped 1 previous similar message [ 4573.215527] Lustre: Skipped 1 previous similar message [ 4576.812726] Lustre: server umount lustre-OST0000 complete [ 4583.400862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.404121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.432842] Lustre: Skipped 1 previous similar message [ 4583.449028] Lustre: Skipped 1 previous similar message [ 4593.634538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.644684] Lustre: Skipped 3 previous similar messages [ 4596.192257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4596.605609] Lustre: server umount lustre-MDT0000 complete [ 4598.781945] LustreError: 138726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.796603] LustreError: 138726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4601.591927] LustreError: 138705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573231 with bad export cookie 10352330219657646331 [ 4601.603054] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.615581] LustreError: 138705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4602.006984] Lustre: server umount lustre-MDT0001 complete [ 4624.730440] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 4627.699968] Key type lgssc unregistered [ 4628.112687] LNet: 145128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4628.126567] LNetError: 145128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4628.150283] LNet: Removed LNI 192.168.204.141@tcp [ 4629.103973] Key type .llcrypt unregistered [ 4629.105644] Key type ._llcrypt unregistered [ 4673.398980] Key type ._llcrypt registered [ 4673.401373] Key type .llcrypt registered [ 4673.496595] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 4691.645867] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 4692.823582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4692.973087] alg: No test for adler32 (adler32-zlib) [ 4694.141856] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 4694.338292] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 4696.000241] Key type lgssc registered [ 4697.343839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4737.067312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4737.104354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.293977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4738.343988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4738.478392] Lustre: lustre-MDT0000: new disk, initializing [ 4738.580295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.611810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4743.876313] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4756.143541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.420591] Lustre: 149286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4756.540907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4756.551726] Lustre: Skipped 1 previous similar message [ 4756.738566] Lustre: lustre-MDT0001: new disk, initializing [ 4756.926856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4757.064798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4757.079289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4763.306126] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4775.224787] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4782.790219] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4790.528679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4790.790579] Lustre: lustre-OST0000: new disk, initializing [ 4790.800857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4790.808260] Lustre: 151258:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4790.968293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4798.540946] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4800.583923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4800.598309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4800.712918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4808.540073] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4813.811437] Lustre: Failing over lustre-OST0000 [ 4814.027844] Lustre: server umount lustre-OST0000 complete [ 4831.712142] Lustre: 147511:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787573445/real 1787573445] req@000000001a9bf5a0 x1874406485821056/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787573461 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4831.748917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.630632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4832.737859] Lustre: 147509:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787573446/real 1787573446] req@000000007349fb00 x1874406485821312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787573462 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4832.765335] Lustre: 147509:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4832.771623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.986571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.016865] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4836.963420] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4837.985105] Lustre: 147511:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787573451/real 1787573451] req@00000000d9e00873 x1874406485823104/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787573467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4839.449313] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4842.129908] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4842.130420] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4843.168198] Lustre: 147511:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787573456/real 1787573456] req@00000000fa39e29a x1874406485824128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787573472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4843.200138] Lustre: 147511:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4843.740075] Lustre: Failing over lustre-MDT0000 [ 4844.003017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4844.015313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.037382] LustreError: 149297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.063684] LustreError: 149297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4844.127493] Lustre: server umount lustre-MDT0000 complete [ 4847.587573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.598288] LustreError: 149293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.626223] Lustre: Skipped 1 previous similar message [ 4847.647112] LustreError: 149293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4848.608541] Lustre: 147510:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787573461/real 1787573461] req@000000005b987fe2 x1874406485825280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787573477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4852.707127] LustreError: 150450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.746554] LustreError: 150450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4857.826588] LustreError: 149297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.850354] LustreError: 149297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4862.953378] LustreError: 150280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.967331] LustreError: 150280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4863.271430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.393271] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.741249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.819262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4867.444282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4868.441748] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4869.108863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4869.118962] Lustre: Skipped 1 previous similar message [ 4869.135168] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4869.174386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4878.305568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.312938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.334644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4879.335388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.962325] Lustre: server umount lustre-OST0000 complete [ 4889.571302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.578273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4889.593652] Lustre: Skipped 1 previous similar message [ 4889.606634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.616610] Lustre: Skipped 2 previous similar messages [ 4894.066074] Lustre: server umount lustre-MDT0000 complete [ 4894.695537] LustreError: 150450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.728168] LustreError: 150450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4898.250117] LustreError: 151257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787573528 with bad export cookie 10414909060543485631 [ 4898.254341] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.270384] LustreError: 151257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4898.916362] Lustre: server umount lustre-MDT0001 complete [ 4913.072958] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 4917.665468] Key type lgssc unregistered [ 4918.246908] LNet: 154916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4918.255055] LNetError: 154916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4918.298250] LNet: Removed LNI 192.168.204.141@tcp [ 4919.895837] Key type .llcrypt unregistered [ 4919.898405] Key type ._llcrypt unregistered [ 4956.731260] Key type ._llcrypt registered [ 4956.732917] Key type .llcrypt registered [ 4956.886982] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 4958.528205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4958.552967] alg: No test for adler32 (adler32-zlib) [ 4959.711634] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 4960.118311] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 4961.836919] Key type lgssc registered [ 4963.326795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4973.196354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4973.230330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.008169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.212440] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 4984.293798] LustreError: 156438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.324359] LustreError: 156438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4989.410514] LustreError: 156439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.994848] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4994.534213] LustreError: 156438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.045766] LustreError: 156467:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4995.060390] LustreError: 156467:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4995.076929] LustreError: 156467:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4995.420320] Lustre: server umount lustre-MDT0000 complete [ 5009.673601] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 5023.287552] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 5061.270445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.674582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5061.735348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5061.828222] Lustre: lustre-MDT0000: new disk, initializing [ 5061.939500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.959287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5066.604382] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5079.628445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.733906] Lustre: 159772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5079.793091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5079.797723] Lustre: Skipped 1 previous similar message [ 5079.870640] Lustre: lustre-MDT0001: new disk, initializing [ 5079.947291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.012825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5080.032353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5085.850589] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5096.514211] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5102.965482] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5110.366975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5110.679215] Lustre: lustre-OST0000: new disk, initializing [ 5110.682959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5110.689365] Lustre: 161743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5110.779077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.506408] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5119.552467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5119.561696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5119.697160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5125.739696] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5129.313295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5130.937732] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:15:59 (1787573759) [ 5412.834067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5412.838350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.845415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5416.928833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5416.931279] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.940100] Lustre: Skipped 1 previous similar message [ 5416.971582] Lustre: Skipped 1 previous similar message [ 5419.097824] Lustre: server umount lustre-MDT0000 complete [ 5422.060253] LustreError: 159784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.083050] LustreError: 159784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5422.980305] LustreError: 160566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574052 with bad export cookie 15486354138417085937 [ 5422.987224] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.994028] LustreError: 160566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5427.169191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.171191] LustreError: 163525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.185201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5427.215651] Lustre: Skipped 1 previous similar message [ 5429.414782] Lustre: server umount lustre-MDT0001 complete [ 5439.388611] Lustre: server umount lustre-OST0000 complete [ 5450.585889] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 5463.010761] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 5504.993219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.300253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5505.366965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5505.479962] Lustre: lustre-MDT0000: new disk, initializing [ 5505.612844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.634561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5510.623466] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5521.876451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.068739] Lustre: 166412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.198057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5522.210412] Lustre: Skipped 1 previous similar message [ 5522.359325] Lustre: lustre-MDT0001: new disk, initializing [ 5522.549108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5522.656452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5522.683572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5528.236782] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5539.883676] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5546.364668] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5554.155333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.428385] Lustre: lustre-OST0000: new disk, initializing [ 5554.431608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5554.437740] Lustre: 168380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5554.542748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.011651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5561.020844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5561.145104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5561.401331] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5573.523305] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5581.366278] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5581.793608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5587.713839] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5587.947340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5592.042330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5592.046892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.100772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5592.112323] Lustre: Skipped 1 previous similar message [ 5596.851643] Lustre: server umount lustre-OST0000 complete [ 5604.321812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.333980] LustreError: Skipped 1 previous similar message [ 5604.342198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.358842] Lustre: Skipped 1 previous similar message [ 5604.370058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.522193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.535154] Lustre: Skipped 3 previous similar messages [ 5617.120285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5617.402480] Lustre: server umount lustre-MDT0000 complete [ 5617.635648] LustreError: 167578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.662331] LustreError: 167578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5622.229548] LustreError: 166404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574252 with bad export cookie 15486354138417244767 [ 5622.232611] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.239890] LustreError: 166404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5622.649335] Lustre: server umount lustre-MDT0001 complete [ 5633.356279] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:24:21 (1787574261) [ 5651.241726] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 5660.981928] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 5698.304037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.767125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5698.808559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5698.864845] Lustre: lustre-MDT0000: new disk, initializing [ 5698.936466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.955188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.124071] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5716.479891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.648768] Lustre: 173140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.907432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5722.167659] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5732.471968] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5738.979156] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.970738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.424882] Lustre: lustre-OST0000: new disk, initializing [ 5748.428341] Lustre: Skipped 1 previous similar message [ 5748.434313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5748.440516] Lustre: Skipped 2 previous similar messages [ 5748.447768] Lustre: 175122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5756.935982] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5758.085594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5758.121431] Lustre: Skipped 1 previous similar message [ 5758.131701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5758.247857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5768.504453] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5778.919622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.920384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.921733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5778.930899] LustreError: Skipped 1 previous similar message [ 5778.966170] Lustre: Skipped 2 previous similar messages [ 5782.850430] Lustre: server umount lustre-OST0000 complete [ 5788.641734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.691913] Lustre: server umount lustre-MDT0000 complete [ 5794.273412] LustreError: 174309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.290600] LustreError: 174309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5797.684098] LustreError: 173133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574427 with bad export cookie 15486354138417245649 [ 5797.688764] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.705805] LustreError: 173133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5798.088988] Lustre: server umount lustre-MDT0001 complete [ 5807.668054] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:27:15 (1787574435) [ 5827.208906] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 5837.523781] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 5876.721513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.036631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.094196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.178598] Lustre: lustre-MDT0000: new disk, initializing [ 5877.349486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.362751] Lustre: Skipped 2 previous similar messages [ 5877.388264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5882.400588] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5894.024561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.141982] Lustre: 179447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5894.434943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5899.124629] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5910.743442] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5917.631911] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.992753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5926.475374] Lustre: 181427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5935.767761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5935.939862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5936.530547] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 5949.152709] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.705883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5961.708305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.719495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.719509] Lustre: Skipped 4 previous similar messages [ 5961.723857] LustreError: Skipped 1 previous similar message [ 5961.739259] Lustre: Skipped 3 previous similar messages [ 5965.464733] Lustre: server umount lustre-OST0000 complete [ 5985.248167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5985.870853] Lustre: server umount lustre-MDT0000 complete [ 5986.292288] LustreError: 179458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.781247] LustreError: 179440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574622 with bad export cookie 15486354138417246734 [ 5992.789995] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.808696] LustreError: 179440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5993.701790] Lustre: server umount lustre-MDT0001 complete [ 6008.609247] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 6012.258612] Key type lgssc unregistered [ 6012.723322] LNet: 183289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6012.731349] LNetError: 183289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6012.762296] LNet: Removed LNI 192.168.204.141@tcp [ 6013.936526] Key type .llcrypt unregistered [ 6013.941512] Key type ._llcrypt unregistered [ 6041.581721] Key type ._llcrypt registered [ 6041.583873] Key type .llcrypt registered [ 6041.722457] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 6042.836277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6042.868233] alg: No test for adler32 (adler32-zlib) [ 6043.983248] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 6044.224826] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 6045.896812] Key type lgssc registered [ 6047.039164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6055.557971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6055.592083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.053720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.108360] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6066.153912] LustreError: 184705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.178878] LustreError: 184705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6071.268907] LustreError: 184706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.825521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.149397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.568218] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6085.130492] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.912444] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6098.713394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.080068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.149691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6105.597456] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6113.432358] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.320675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.324995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.333511] Lustre: Skipped 1 previous similar message [ 6137.349103] Lustre: Skipped 1 previous similar message [ 6142.435895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.445931] Lustre: Skipped 1 previous similar message [ 6142.714938] Lustre: server umount lustre-OST0000 complete [ 6147.557678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.566382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.575904] Lustre: Skipped 1 previous similar message [ 6152.686897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.694835] Lustre: Skipped 2 previous similar messages [ 6157.800773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.810929] Lustre: Skipped 1 previous similar message [ 6161.376851] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6161.533673] Lustre: server umount lustre-MDT0000 complete [ 6162.916641] LustreError: 184706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.941470] LustreError: 184706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6165.527598] LustreError: 185441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574795 with bad export cookie 14189008067869525222 [ 6165.531179] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.542376] LustreError: 185441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6165.880565] Lustre: server umount lustre-MDT0001 complete [ 6178.557472] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 6181.724622] Key type lgssc unregistered [ 6182.029626] LNet: 188270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6182.041831] LNetError: 188270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6182.068618] LNet: Removed LNI 192.168.204.141@tcp [ 6182.978293] Key type .llcrypt unregistered [ 6182.982077] Key type ._llcrypt unregistered [ 6211.008308] Key type ._llcrypt registered [ 6211.011197] Key type .llcrypt registered [ 6211.184355] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 6212.782240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6212.875911] alg: No test for adler32 (adler32-zlib) [ 6214.078904] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 6214.364821] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 6216.088209] Key type lgssc registered [ 6217.326797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6226.148324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6226.182755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.725055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.600695] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6237.155447] LustreError: 189688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.170144] LustreError: 189688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6242.277743] LustreError: 189689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.098684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.566668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6248.469051] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6257.478777] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6264.401024] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6271.646570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.072756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.146881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6278.821764] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6287.976583] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6294.917872] Lustre: lustre-OST0000: Not available for connect from 192.168.204.41@tcp (stopping) [ 6299.111297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6299.111654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.130472] LustreError: Skipped 1 previous similar message [ 6299.144472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6299.166407] Lustre: Skipped 1 previous similar message [ 6300.710944] Lustre: server umount lustre-OST0000 complete [ 6309.309710] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a04c93a7000.ost_server_uuid 50 [ 6311.515751] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6329.835472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.859429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.860186] Lustre: Skipped 2 previous similar messages [ 6329.878620] Lustre: Skipped 2 previous similar messages [ 6332.374177] Lustre: server umount lustre-MDT0000 complete [ 6334.950332] LustreError: 189689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.979969] LustreError: 189689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6336.671177] LustreError: 189667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574966 with bad export cookie 8044782210413247536 [ 6336.686764] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.223635] Lustre: server umount lustre-MDT0001 complete [ 6346.681947] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:36:14 (1787574974) [ 6353.985175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6354.591287] LustreError: 193606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.655704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.819307] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6360.043229] LustreError: 193607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.665646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.033479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6372.084678] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6379.891615] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.845358] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.882411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.225850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.353167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6400.266108] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6409.151950] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.528695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.803814] Lustre: lustre-OST0001: new disk, initializing [ 6417.808652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6417.817406] Lustre: 196448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6417.929220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.541683] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6427.724167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6427.730169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6427.822686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6436.403786] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6440.420765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.432754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.456522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.491656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6443.504638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.531211] Lustre: server umount lustre-OST0000 complete [ 6447.482604] LustreError: 195431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.505694] LustreError: 195431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6452.130724] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6452.337568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6456.723980] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6456.937925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6463.981231] LustreError: 195431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.002399] LustreError: 195431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6474.210165] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.211653] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.214642] LustreError: Skipped 1 previous similar message [ 6474.231766] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6474.234556] Lustre: Skipped 3 previous similar messages [ 6476.045705] Lustre: server umount lustre-OST0001 complete [ 6484.450959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6484.454062] Lustre: Skipped 3 previous similar messages [ 6485.427664] Lustre: server umount lustre-MDT0000 complete [ 6488.924215] LustreError: 193587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575118 with bad export cookie 8044782210413248607 [ 6488.929868] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6488.930709] LustreError: 193587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6489.225896] Lustre: server umount lustre-MDT0001 complete [ 6513.894447] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:39:02 (1787575142) [ 6520.283099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.595644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6520.642198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6520.899763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.003536] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6533.965114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.131214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6534.189459] Lustre: 199890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6539.378357] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6547.352547] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6552.428859] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6558.749267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6558.928140] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6564.809777] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6566.056868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6572.422996] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.055477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6581.229163] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6588.659677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6588.897149] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6598.212718] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.697349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6601.701853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.710529] Lustre: Skipped 3 previous similar messages [ 6601.714880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.492611] Lustre: server umount lustre-OST0000 complete [ 6609.260946] LustreError: 203444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.280947] LustreError: 203444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6624.737166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6624.739770] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.757316] LustreError: Skipped 2 previous similar messages [ 6624.792483] Lustre: Skipped 2 previous similar messages [ 6629.943098] Lustre: server umount lustre-OST0001 complete [ 6634.985026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6635.001508] Lustre: Skipped 7 previous similar messages [ 6640.938366] Lustre: server umount lustre-MDT0000 complete [ 6646.050759] LustreError: 199882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575275 with bad export cookie 8044782210413250014 [ 6646.060675] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.068168] LustreError: 199882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6646.602983] Lustre: server umount lustre-MDT0001 complete [ 6676.273674] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:41:44 (1787575304) [ 6695.146467] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 6708.375410] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 6751.244754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.449702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6751.470967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6751.537962] Lustre: lustre-MDT0000: new disk, initializing [ 6751.606952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6751.611903] Lustre: Skipped 3 previous similar messages [ 6751.624183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6756.924501] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6769.049560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.207109] Lustre: 207987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6769.300833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6769.304645] Lustre: Skipped 1 previous similar message [ 6769.409402] Lustre: lustre-MDT0001: new disk, initializing [ 6769.558881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6769.574687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6774.379732] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6784.919665] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6791.670890] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6799.906786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.342488] Lustre: lustre-OST0000: new disk, initializing [ 6800.347870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6800.363477] Lustre: 209954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6805.649845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6805.673618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6805.801362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6808.511394] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6819.879901] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.978105] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6826.299654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6832.401644] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6832.665449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6836.707303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.709866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.724153] LustreError: Skipped 1 previous similar message [ 6836.725051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.725056] Lustre: Skipped 2 previous similar messages [ 6836.757464] Lustre: Skipped 3 previous similar messages [ 6841.741375] Lustre: server umount lustre-OST0000 complete [ 6852.067039] LustreError: 207992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.085312] LustreError: 207992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 6852.561974] Lustre: server umount lustre-MDT0000 complete [ 6857.782356] LustreError: 208781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575487 with bad export cookie 8044782210413251799 [ 6857.788812] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6857.802475] LustreError: 208781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6858.262660] Lustre: server umount lustre-MDT0001 complete [ 6868.827681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.767906] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6884.909900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.352881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6885.361251] Lustre: Skipped 3 previous similar messages [ 6890.829235] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6899.971581] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.863428] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6916.454763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.789356] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 6934.936591] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.454907] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6942.608944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6948.902289] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6949.273754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6954.977360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.016489] Lustre: Skipped 3 previous similar messages [ 6965.222508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.228085] Lustre: Skipped 8 previous similar messages [ 6966.242497] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6966.465858] Lustre: server umount lustre-OST0000 complete [ 6975.065502] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6980.463615] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7010.580325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.277913] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7020.339644] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7020.361401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7020.369197] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7020.396873] Lustre: Skipped 1 previous similar message [ 7030.731114] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.695222] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7040.993568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7041.014737] LustreError: Skipped 1 previous similar message [ 7044.615375] Lustre: server umount lustre-OST0000 complete [ 7055.274570] Lustre: server umount lustre-MDT0000 complete [ 7056.362612] LustreError: 212174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.409251] LustreError: 212174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7059.653293] LustreError: 216288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575689 with bad export cookie 8044782210413252681 [ 7059.658313] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.664037] LustreError: 216288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7060.244315] Lustre: server umount lustre-MDT0001 complete [ 7073.063825] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:48:20 (1787575700) [ 7082.867728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.524585] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7099.593615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.141880] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7113.403710] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7120.598690] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7129.539977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7137.228530] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7146.915946] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7153.643664] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7153.944241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7159.956909] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7160.299750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7170.327106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.777162] Lustre: lustre-OST0001: new disk, initializing [ 7170.787944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7170.810676] Lustre: 221518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7171.046144] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7171.062871] Lustre: Skipped 5 previous similar messages [ 7178.440275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7178.464765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7178.732395] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7179.294616] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7190.620937] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7196.388864] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7196.688189] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7202.505298] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7202.777143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7209.443873] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7209.449541] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.469025] LustreError: Skipped 2 previous similar messages [ 7209.496509] Lustre: Skipped 5 previous similar messages [ 7211.205340] Lustre: server umount lustre-OST0001 complete [ 7217.794889] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7218.099695] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7223.903455] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7224.150965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7253.396815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7259.121396] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7259.134190] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7259.135363] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7262.273670] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7272.945551] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7275.632228] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7279.600705] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7279.613536] Lustre: Skipped 8 previous similar messages [ 7284.245466] Lustre: server umount lustre-OST0001 complete [ 7308.063602] Lustre: server umount lustre-OST0000 complete [ 7317.682166] Lustre: server umount lustre-MDT0000 complete [ 7320.546383] LustreError: 219599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.566242] LustreError: 219599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7321.275077] LustreError: 218128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575951 with bad export cookie 8044782210413253647 [ 7321.277739] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7321.286334] LustreError: 218128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7327.760140] Lustre: server umount lustre-MDT0001 complete [ 7354.238560] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:53:02 (1787575982) [ 7363.064076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.550359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7363.624077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7369.685931] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7379.816681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.039223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7380.081127] Lustre: 226766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.259619] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7396.890352] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7404.877734] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7413.100744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.345652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7420.693270] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7430.513528] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7442.560188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7442.699899] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7450.543267] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7460.260283] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7462.499704] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7468.700281] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7468.935800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7474.307972] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7474.611820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7482.075027] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a04d17e5000.ost_server_uuid 50 [ 7484.052604] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a04d17e5000.ost_server_uuid in IDLE state after 0 sec [ 7485.300661] Lustre: Permanently deactivating lustre-OST0001 [ 7485.312338] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7490.701771] Lustre: Permanently reactivating lustre-OST0001 [ 7490.725827] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7490.736477] Lustre: Skipped 2 previous similar messages [ 7497.189351] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7497.195362] LustreError: Skipped 5 previous similar messages [ 7497.198150] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.205539] Lustre: Skipped 6 previous similar messages [ 7502.113857] Lustre: server umount lustre-OST0001 complete [ 7512.910746] Lustre: server umount lustre-OST0000 complete [ 7523.574571] Lustre: server umount lustre-MDT0000 complete [ 7527.317712] LustreError: 226759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576157 with bad export cookie 8044782210413254928 [ 7527.326873] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.332548] LustreError: 226759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7527.772919] Lustre: server umount lustre-MDT0001 complete [ 7554.596751] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:56:22 (1787576182) [ 7562.813149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.102860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7563.127302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7563.133301] Lustre: Skipped 2 previous similar messages [ 7568.011666] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7578.069958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.134812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7578.173499] Lustre: 233467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7583.229185] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7590.747376] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7597.487582] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7604.874587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7605.101748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7605.126213] Lustre: Permanently deactivating lustre-OST0000 [ 7605.137272] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7611.970618] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7615.004946] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7620.913639] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.680927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.793648] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7635.664940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7636.112674] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7644.462812] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7646.568355] Lustre: Permanently reactivating lustre-OST0000 [ 7646.586948] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7646.592719] Lustre: Skipped 2 previous similar messages [ 7653.947856] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7653.948516] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 000000001512e0aa, cur 1787576283 deadline 1787576274 last 1787576244 [ 7653.963642] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7653.994022] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7654.001777] Lustre: Skipped 1 previous similar message [ 7654.069835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7660.073848] Lustre: server umount lustre-OST0001 complete [ 7670.535787] Lustre: server umount lustre-OST0000 complete [ 7681.696035] Lustre: server umount lustre-MDT0000 complete [ 7685.730924] LustreError: 233459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576315 with bad export cookie 8044782210413256608 [ 7685.741388] LustreError: 233459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7685.741770] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.048245] Lustre: server umount lustre-MDT0001 complete [ 7698.116108] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 7701.155325] Key type lgssc unregistered [ 7701.532121] LNet: 238330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.561206] LNetError: 238330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.584411] LNet: Removed LNI 192.168.204.141@tcp [ 7702.577708] Key type .llcrypt unregistered [ 7702.586274] Key type ._llcrypt unregistered [ 7727.342567] Key type ._llcrypt registered [ 7727.350633] Key type .llcrypt registered [ 7727.499490] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 7729.504241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7729.552518] alg: No test for adler32 (adler32-zlib) [ 7730.548448] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 7730.935817] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 7732.608219] Key type lgssc registered [ 7734.169573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7744.881283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7744.901185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.444251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.804854] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7756.772796] LustreError: 239795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.792036] LustreError: 239795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7760.145811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.292550] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7760.316835] Lustre: 239785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7760.329648] Lustre: Permanently deactivating lustre-MDT0001 [ 7760.353806] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7760.692298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.202931] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7766.056799] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7773.057995] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7779.693640] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7787.572676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.891181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7793.001417] LustreError: 241636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.014981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7794.417249] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7798.253828] LustreError: 241636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.363355] LustreError: 241930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.899702] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7810.553181] LustreError: 241636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.590328] LustreError: 241636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7814.828795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.138933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7820.301469] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7820.335822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7824.102958] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7833.811550] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7838.217259] Lustre: Permanently reactivating lustre-MDT0001 [ 7838.246164] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7838.251230] Lustre: Skipped 2 previous similar messages [ 7843.740383] Lustre: lustre-MDT0001: Client 36c43559-7a54-41ed-b99a-8d451ffebcda (at 192.168.204.41@tcp) reconnecting [ 7844.461609] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.476565] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7844.484290] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7844.498370] LustreError: 243375:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7844.502077] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7844.511679] LustreError: 243375:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7844.511818] LustreError: 243375:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.141@tcp: cfg command failed: rc = -108 [ 7844.511840] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7844.511840] [ 7844.521048] LustreError: 239787:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7850.711649] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7852.544739] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7859.113848] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7859.389448] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7865.572358] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7865.936514] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7869.979674] Lustre: Permanently deactivating lustre-MDT0001 [ 7870.003264] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7870.011824] Lustre: Skipped 2 previous similar messages [ 7879.783988] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7891.939433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.950396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7891.964336] Lustre: Skipped 3 previous similar messages [ 7891.976481] Lustre: Skipped 2 previous similar messages [ 7895.812464] Lustre: server umount lustre-MDT0000 complete [ 7897.065830] LustreError: 239794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.088989] LustreError: 239794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7899.823631] LustreError: 244294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576529 with bad export cookie 10206826291664617833 [ 7899.824227] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.836869] LustreError: 244294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7900.161685] Lustre: server umount lustre-MDT0001 complete [ 7906.561935] Lustre: server umount lustre-OST0000 complete [ 7917.285578] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:02:25 (1787576545) [ 7928.312128] Lustre: server umount lustre-OST0001 complete [ 7934.606192] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 7945.977850] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 7982.561469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.840863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.848348] Lustre: Skipped 2 previous similar messages [ 7982.870338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7982.985777] Lustre: lustre-MDT0000: new disk, initializing [ 7983.060416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.079627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7988.021455] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 7998.798715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.949943] Lustre: 248117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.003484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7999.007720] Lustre: Skipped 1 previous similar message [ 7999.144033] Lustre: lustre-MDT0001: new disk, initializing [ 7999.273111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.356090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7999.364699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8004.345665] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8014.433993] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.562424] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8027.601317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.864401] Lustre: lustre-OST0000: new disk, initializing [ 8027.868735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8027.880690] Lustre: 250088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8027.958123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.181545] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8034.382182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8034.390768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8034.521809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8044.628736] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8047.476044] LustreError: 248123:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8056.050810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.222664] Lustre: lustre-OST0001: new disk, initializing [ 8056.228267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8056.234397] Lustre: 251203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8057.488168] LustreError: 248123:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8057.527061] LustreError: 251145:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8062.312194] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8064.576402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8064.599541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8064.675405] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8067.600449] LustreError: 251145:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8072.355558] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.264885] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8078.216173] Lustre: lustre-OST0001: Not available for connect from 192.168.204.41@tcp (stopping) [ 8078.228838] Lustre: Skipped 1 previous similar message [ 8080.358556] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.360806] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.382323] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8080.399814] Lustre: Skipped 1 previous similar message [ 8082.894396] Lustre: server umount lustre-OST0001 complete [ 8083.325643] LustreError: 250072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.844373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.851441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.857151] Lustre: Skipped 2 previous similar messages [ 8100.882701] Lustre: Skipped 1 previous similar message [ 8105.957118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8105.974022] Lustre: Skipped 1 previous similar message [ 8106.654592] Lustre: server umount lustre-OST0000 complete [ 8111.079804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.090968] Lustre: Skipped 1 previous similar message [ 8116.196088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.206756] Lustre: Skipped 3 previous similar messages [ 8117.171317] Lustre: server umount lustre-MDT0000 complete [ 8121.312854] LustreError: 248127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.349476] LustreError: 248127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8121.797499] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576751 with bad export cookie 10206826291664620430 [ 8121.802658] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8121.809885] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8136.160952] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8136.510762] Lustre: server umount lustre-MDT0001 complete [ 8150.111659] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 8152.995308] Key type lgssc unregistered [ 8153.424892] LNet: 253240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8153.436379] LNetError: 253240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8153.456801] LNet: Removed LNI 192.168.204.141@tcp [ 8154.635827] Key type .llcrypt unregistered [ 8154.639076] Key type ._llcrypt unregistered [ 8195.350819] Key type ._llcrypt registered [ 8195.353557] Key type .llcrypt registered [ 8195.466629] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 8196.521354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8196.557335] alg: No test for adler32 (adler32-zlib) [ 8197.626532] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8197.911547] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 8199.632304] Key type lgssc registered [ 8200.864631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8208.862088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8208.879322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.157619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8210.178894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8210.313311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.619683] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8224.297761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.461804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8224.505798] Lustre: 255451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8225.114109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.319431] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8237.559867] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8244.971887] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.044654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.354513] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8252.708403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.793429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8261.080655] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8271.481781] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8325.605099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.607647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.623956] LustreError: Skipped 1 previous similar message [ 8325.637451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8325.655774] Lustre: Skipped 1 previous similar message [ 8330.725876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.737969] Lustre: Skipped 1 previous similar message [ 8331.574809] Lustre: server umount lustre-OST0000 complete [ 8336.185534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8351.384618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.403075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.456635] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8355.476923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8355.492107] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8355.523314] Lustre: Skipped 1 previous similar message [ 8364.187490] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8373.803298] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8417.261229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8417.276594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.281669] LustreError: Skipped 1 previous similar message [ 8417.282042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8417.326113] Lustre: Skipped 2 previous similar messages [ 8419.777407] Lustre: server umount lustre-OST0000 complete [ 8424.929467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8424.945510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.966468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.979496] Lustre: Skipped 1 previous similar message [ 8427.491429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8430.743518] Lustre: server umount lustre-MDT0000 complete [ 8434.876784] LustreError: 255443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577064 with bad export cookie 15591620353496935089 [ 8434.879705] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8434.885135] LustreError: 255443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8435.209720] Lustre: server umount lustre-MDT0001 complete [ 8450.923919] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 8454.081395] Key type lgssc unregistered [ 8454.429727] LNet: 260879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8454.437953] LNetError: 260879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8454.456789] LNet: Removed LNI 192.168.204.141@tcp [ 8455.315372] Key type .llcrypt unregistered [ 8455.319980] Key type ._llcrypt unregistered [ 8479.794842] Key type ._llcrypt registered [ 8479.798809] Key type .llcrypt registered [ 8479.927486] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 8480.786464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8480.813452] alg: No test for adler32 (adler32-zlib) [ 8481.775517] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8481.953580] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 8483.640358] Key type lgssc registered [ 8484.889433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8493.158865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8493.180470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.577486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8499.186461] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8504.803175] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.823026] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8507.204583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.576155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8511.500105] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8518.703620] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8523.966853] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.900331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8534.165228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.357327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8543.389241] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8553.275519] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.584879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8580.623345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.676319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8583.651509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8583.661196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8583.694331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.103707] Lustre: server umount lustre-OST0000 complete [ 8593.896115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.907722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8593.920266] Lustre: Skipped 1 previous similar message [ 8593.944984] Lustre: Skipped 2 previous similar messages [ 8596.931423] Lustre: server umount lustre-MDT0000 complete [ 8599.015585] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.065434] LustreError: 262299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8601.206760] LustreError: 262278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577231 with bad export cookie 13783875769903314857 [ 8601.215611] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.237502] LustreError: 262278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8601.567693] Lustre: server umount lustre-MDT0001 complete [ 8616.678920] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 8619.953740] Key type lgssc unregistered [ 8620.324426] LNet: 266608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8620.334542] LNetError: 266608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8620.361158] LNet: Removed LNI 192.168.204.141@tcp [ 8621.426993] Key type .llcrypt unregistered [ 8621.428970] Key type ._llcrypt unregistered [ 8645.921197] Key type ._llcrypt registered [ 8645.922725] Key type .llcrypt registered [ 8646.044812] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 8647.208655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8647.293367] alg: No test for adler32 (adler32-zlib) [ 8648.346395] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8648.556989] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 8650.218734] Key type lgssc registered [ 8651.795085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8660.904265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8660.952269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8662.571321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.921139] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8672.746175] LustreError: 267831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8672.768575] LustreError: 267831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8676.851633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.275440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8682.107183] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8690.920377] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8696.866275] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8704.242761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8704.876563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.980937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8712.881627] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8723.649160] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8737.765138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8737.769118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.791715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8737.799956] Lustre: Skipped 1 previous similar message [ 8737.815123] Lustre: Skipped 1 previous similar message [ 8740.299359] Lustre: server umount lustre-OST0000 complete [ 8748.007707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.037729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8748.068424] Lustre: Skipped 1 previous similar message [ 8753.122500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8753.141279] Lustre: Skipped 1 previous similar message [ 8757.223092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.227904] Lustre: Skipped 1 previous similar message [ 8759.266907] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8759.701512] Lustre: server umount lustre-MDT0000 complete [ 8762.337692] LustreError: 268559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.356863] LustreError: 268559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8765.113129] LustreError: 267813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577394 with bad export cookie 5030149565798036313 [ 8765.119563] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.164605] LustreError: 267813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8765.760882] Lustre: server umount lustre-MDT0001 complete [ 8779.124302] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 8782.952496] Key type lgssc unregistered [ 8783.276059] LNet: 271515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8783.287236] LNetError: 271515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8783.313855] LNet: Removed LNI 192.168.204.141@tcp [ 8784.364609] Key type .llcrypt unregistered [ 8784.366557] Key type ._llcrypt unregistered [ 8814.190573] Key type ._llcrypt registered [ 8814.201283] Key type .llcrypt registered [ 8814.360653] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 8816.030715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8816.084815] alg: No test for adler32 (adler32-zlib) [ 8817.096713] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8817.284193] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 8818.969239] Key type lgssc registered [ 8820.359949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8828.993157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8829.031419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.473663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8835.786283] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8840.679182] LustreError: 272936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.706408] LustreError: 272936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8845.705969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.799863] LustreError: 272937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.158031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8851.195192] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8860.062377] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8866.798542] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8875.449808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.010198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.085847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8883.674142] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 8891.993258] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8917.986471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8917.999457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.014544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8921.064838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8921.065415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8921.089096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.293080] Lustre: server umount lustre-OST0000 complete [ 8928.236195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8928.255642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.294093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.308968] Lustre: Skipped 1 previous similar message [ 8931.311809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8931.316704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.405499] Lustre: server umount lustre-MDT0000 complete [ 8936.425408] LustreError: 272937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.453305] LustreError: 272937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8937.458124] LustreError: 272917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577567 with bad export cookie 16662048312548359786 [ 8937.475978] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8937.478390] LustreError: 272917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8937.936360] Lustre: server umount lustre-MDT0001 complete [ 8950.107914] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 8953.293481] Key type lgssc unregistered [ 8953.693495] LNet: 277246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.706621] LNetError: 277246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8954.737043] LNet: Removed LNI 192.168.204.141@tcp [ 8955.458685] Key type .llcrypt unregistered [ 8955.460742] Key type ._llcrypt unregistered [ 8977.885323] Key type ._llcrypt registered [ 8977.889779] Key type .llcrypt registered [ 8978.035384] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 8979.565743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8979.598127] alg: No test for adler32 (adler32-zlib) [ 8980.719865] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 8980.965272] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 8982.664283] Key type lgssc registered [ 8983.813679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8992.150590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8992.173776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.834337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8998.288893] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9004.047140] LustreError: 278484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9004.084102] LustreError: 278484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9008.579756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.076502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9014.563483] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9024.578381] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9031.375178] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9039.524981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.051071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9046.516783] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9054.879968] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9063.603996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9079.272154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.279550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.320772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9079.326702] Lustre: Skipped 1 previous similar message [ 9079.350332] Lustre: Skipped 1 previous similar message [ 9084.391171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.405363] Lustre: Skipped 1 previous similar message [ 9085.288314] Lustre: server umount lustre-OST0000 complete [ 9091.042813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9091.052170] LustreError: Skipped 1 previous similar message [ 9091.058361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9091.076886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9094.630851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9094.643258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.855118] Lustre: server umount lustre-MDT0000 complete [ 9099.746325] LustreError: 278482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9099.776729] LustreError: 278482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9100.366312] LustreError: 278451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577730 with bad export cookie 11830461105316568852 [ 9100.371061] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.387100] LustreError: 278451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9100.740629] Lustre: server umount lustre-MDT0001 complete [ 9113.791390] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 9116.548657] Key type lgssc unregistered [ 9116.820707] LNet: 282216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9116.828625] LNetError: 282216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9116.842777] LNet: Removed LNI 192.168.204.141@tcp [ 9117.695135] Key type .llcrypt unregistered [ 9117.701222] Key type ._llcrypt unregistered [ 9129.979067] Key type ._llcrypt registered [ 9129.980658] Key type .llcrypt registered [ 9130.106612] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9151.214428] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 9167.530434] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 9168.896564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9169.049528] alg: No test for adler32 (adler32-zlib) [ 9170.062278] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 9170.278448] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 9171.960228] Key type lgssc registered [ 9173.407886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9217.345738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9217.373587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9218.793050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9218.846441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9218.914572] Lustre: lustre-MDT0000: new disk, initializing [ 9218.997623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9219.037152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9223.899796] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9235.926194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9236.043771] Lustre: 285787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9236.104194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9236.107808] Lustre: Skipped 1 previous similar message [ 9236.157928] Lustre: lustre-MDT0001: new disk, initializing [ 9236.223554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9236.250160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9236.255873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9242.370276] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9256.939953] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9265.394595] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9274.372914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9274.729435] Lustre: lustre-OST0000: new disk, initializing [ 9274.737201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9274.753491] Lustre: 287758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9274.917860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9281.807574] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9282.157566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9282.166803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9282.274932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9292.993680] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9298.768408] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9299.096974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9304.354261] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9304.644519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9308.132482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.134965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9308.138272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9308.147246] Lustre: Skipped 1 previous similar message [ 9313.234201] Lustre: server umount lustre-OST0000 complete [ 9318.370312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9318.375383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.380093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9318.396392] Lustre: Skipped 1 previous similar message [ 9318.410830] Lustre: Skipped 2 previous similar messages [ 9323.490418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.501901] Lustre: Skipped 1 previous similar message [ 9323.883510] Lustre: server umount lustre-MDT0000 complete [ 9327.731785] LustreError: 286590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577957 with bad export cookie 6808682857791339545 [ 9327.737879] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9327.752034] LustreError: 286590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9328.330270] Lustre: server umount lustre-MDT0001 complete [ 9337.971523] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:26:05 (1787577965) [ 9345.474986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.107646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9350.962863] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9351.144385] LustreError: 290170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9351.171216] LustreError: 290170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9355.244089] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.360220] LustreError: 290170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.239609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.885767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9369.514395] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9379.090667] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9386.769127] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9396.258182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9396.615696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9403.252469] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9410.904783] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9427.430568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9427.439423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.446126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9427.455763] Lustre: Skipped 1 previous similar message [ 9427.468218] Lustre: Skipped 1 previous similar message [ 9432.556995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9432.564771] Lustre: Skipped 1 previous similar message [ 9433.433626] Lustre: server umount lustre-OST0000 complete [ 9439.714229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9439.732667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.810541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9442.826407] Lustre: Skipped 2 previous similar messages [ 9444.576498] Lustre: server umount lustre-MDT0000 complete [ 9447.907241] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.932677] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9448.645711] LustreError: 292015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578078 with bad export cookie 6808682857791340427 [ 9448.651039] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9448.664194] LustreError: 292015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9449.218133] Lustre: server umount lustre-MDT0001 complete [ 9463.774714] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [ 9466.578019] Key type lgssc unregistered [ 9466.865835] LNet: 293733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9466.881330] LNetError: 293733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9466.900907] LNet: Removed LNI 192.168.204.141@tcp [ 9467.791607] Key type .llcrypt unregistered [ 9467.795634] Key type ._llcrypt unregistered [ 9497.044439] Key type ._llcrypt registered [ 9497.046752] Key type .llcrypt registered [ 9497.172609] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 9513.833595] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 9516.285896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9516.893656] alg: No test for adler32 (adler32-zlib) [ 9518.111730] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [ 9518.474584] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [ 9520.168916] Key type lgssc registered [ 9521.876643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9576.628102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9576.679150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9577.995266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9578.061317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9578.182270] Lustre: lustre-MDT0000: new disk, initializing [ 9578.322996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.343323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9584.632888] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9598.463617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.612759] Lustre: 297561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9598.766494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9598.769256] Lustre: Skipped 1 previous similar message [ 9598.827448] Lustre: lustre-MDT0001: new disk, initializing [ 9598.932673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9598.979450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9598.992458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9605.978266] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9619.938160] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9628.549826] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9639.333559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9639.685626] Lustre: lustre-OST2710: new disk, initializing [ 9639.690394] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9639.699293] Lustre: 299535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9639.791202] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9646.311525] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9649.758281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9649.774889] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9649.895607] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9656.765747] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9665.572129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9665.844604] Lustre: lustre-OST03e8: new disk, initializing [ 9665.848353] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9665.856147] Lustre: 300598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9665.930432] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9674.215691] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9675.498038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9675.513889] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9675.751699] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9684.040975] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9694.077948] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9694.364409] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9700.155442] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9700.456497] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9706.725856] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9706.988417] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9712.765197] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9713.095972] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9720.712824] Lustre: server umount lustre-MDT0000 complete [ 9721.836658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.838936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9721.848270] Lustre: Skipped 1 previous similar message [ 9725.242991] LustreError: 299534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578355 with bad export cookie 11189861415529410473 [ 9725.257311] LustreError: 299534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9725.260517] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9725.745214] Lustre: server umount lustre-MDT0001 complete [ 9740.664543] Lustre: server umount lustre-OST2710 complete [ 9743.136989] Lustre: 295361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578356/real 1787578356] req@00000000a9aa93a2 x1874411544190848/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787578372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9743.190484] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.213305] Lustre: Skipped 2 previous similar messages [ 9744.815723] Lustre: server umount lustre-OST03e8 complete [ 9765.278105] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [ 9775.996785] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 9819.479818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.853727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9819.911266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9820.042387] Lustre: lustre-MDT0000: new disk, initializing [ 9820.161430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.201613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9825.382510] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9837.934707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.093672] Lustre: 305849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9838.138606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9838.141294] Lustre: Skipped 1 previous similar message [ 9838.189697] Lustre: lustre-MDT0001: new disk, initializing [ 9838.251331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9838.281371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9838.294692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9844.313795] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9859.981090] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9870.829866] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9881.390040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9881.841817] Lustre: lustre-OST0000: new disk, initializing [ 9881.851233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9881.860778] Lustre: 307824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9882.022984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9888.072447] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [ 9888.892255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9888.901166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9889.018424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9898.731753] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9904.414623] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9904.663948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9910.155497] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9910.524729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9914.854853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9914.858048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9914.868406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9914.873459] Lustre: Skipped 1 previous similar message [ 9914.890525] LustreError: Skipped 1 previous similar message [ 9919.988137] LustreError: 308207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9920.017193] LustreError: 308207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9920.241518] Lustre: server umount lustre-OST0000 complete [ 9930.219346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9930.222736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9930.236256] Lustre: Skipped 1 previous similar message [ 9930.244568] Lustre: Skipped 2 previous similar messages [ 9932.360413] Lustre: server umount lustre-MDT0000 complete [ 9935.332468] LustreError: 305863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9935.373516] LustreError: 305863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9937.986596] LustreError: 305842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578567 with bad export cookie 11189861415529412153 [ 9937.990120] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9938.000093] LustreError: 305842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9938.543610] Lustre: server umount lustre-MDT0001 complete [ 9948.510632] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:36:16 (1787578576) [ 9950.014665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9952.013684] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:36:20 (1787578580) [ 9955.863396] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [ 9994.883885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.087472] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9995.102358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9995.266967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10000.976562] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10010.372146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.497573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10010.521566] Lustre: 311683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10015.686250] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10039.170658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.366946] Lustre: Denying initial registration attempt from nid 192.168.204.141@tcp, specified as failover [10039.373591] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10039.383718] LustreError: 313136:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [10039.389433] LustreError: 313136:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10039.394374] LustreError: 313136:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10039.470680] Lustre: server umount lustre-OST0000 complete [10039.473993] LustreError: 313136:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10052.066439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10052.074248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.094400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.210274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.217254] Lustre: Skipped 3 previous similar messages [10063.328400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10063.332967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.359980] Lustre: Skipped 1 previous similar message [10063.557576] Lustre: server umount lustre-MDT0000 complete [10067.980297] LustreError: 311674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578697 with bad export cookie 11189861415529413035 [10067.994060] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.467063] Lustre: server umount lustre-MDT0001 complete [10082.311296] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [10086.963635] Key type lgssc unregistered [10087.394707] LNet: 314263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10087.411965] LNetError: 314263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10087.443772] LNet: Removed LNI 192.168.204.141@tcp [10088.616390] Key type .llcrypt unregistered [10088.618805] Key type ._llcrypt unregistered [10104.656679] Key type ._llcrypt registered [10104.659632] Key type .llcrypt registered [10104.770582] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [10105.952205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10106.015623] alg: No test for adler32 (adler32-zlib) [10107.100239] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [10107.269207] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [10108.952293] Key type lgssc registered [10110.415782] Lustre: Echo OBD driver; http://www.lustre.org/ [10167.356646] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [10178.509866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10178.559418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10179.870198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10179.906132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10180.227188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10185.366968] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10194.184864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.347946] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10194.381509] Lustre: 317033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10194.608478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10199.694623] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10208.017761] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10215.164452] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10237.680770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10238.015029] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10238.353170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10244.937470] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10253.205073] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10259.427349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10259.441487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.476221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10259.942602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10259.948798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10261.847811] Lustre: server umount lustre-OST0000 complete [10266.597115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10266.612637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10266.636227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10266.641996] Lustre: Skipped 1 previous similar message [10270.179786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.195175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.465136] Lustre: server umount lustre-MDT0000 complete [10279.192327] LustreError: 317024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578909 with bad export cookie 11720888157146847990 [10279.197061] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10279.198648] LustreError: 317024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10279.780762] Lustre: server umount lustre-MDT0001 complete [10294.207928] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [10299.247915] Key type lgssc unregistered [10299.568591] LNet: 320609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10299.581028] LNetError: 320609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10299.599704] LNet: Removed LNI 192.168.204.141@tcp [10301.404522] Key type .llcrypt unregistered [10301.413349] Key type ._llcrypt unregistered [10329.815228] Key type ._llcrypt registered [10329.817381] Key type .llcrypt registered [10329.938956] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [10331.371096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.395159] alg: No test for adler32 (adler32-zlib) [10332.478626] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [10332.753104] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [10334.424356] Key type lgssc registered [10335.732791] Lustre: Echo OBD driver; http://www.lustre.org/ [10343.826694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10343.847657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.396583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.734455] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10355.684405] LustreError: 322028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.708075] LustreError: 322028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10360.075550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10360.678429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10366.200739] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10377.077928] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10384.371841] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10395.328484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.973108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10405.660807] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10415.514914] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10427.365466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10427.377456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.405200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.295495] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) [10428.917674] Lustre: server umount lustre-MDT0000 complete [10430.956705] LustreError: 322028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.959711] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10433.404560] LustreError: 322023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.431463] LustreError: 322023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10433.585747] LustreError: 323873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579063 with bad export cookie 5384799520838112693 [10433.593056] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10433.606101] LustreError: 323873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10436.073977] LustreError: 323153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.078368] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.126617] Lustre: Skipped 1 previous similar message [10436.135653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10438.542426] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.41@tcp (stopping) [10438.550131] Lustre: Skipped 1 previous similar message [10440.111615] Lustre: server umount lustre-MDT0001 complete [10450.744225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10463.967520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.075703] LustreError: 321336:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b2a1b352 x1874412397691520/t0(0) o250->MGC192.168.204.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10479.949423] LustreError: 323861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10479.971802] LustreError: 323861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10480.039105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.283937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10486.229267] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10488.291557] LustreError: 325623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10488.322136] LustreError: 325623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10493.434135] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10495.787775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.352610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10501.637843] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10501.781925] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10509.691909] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10511.775157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10518.949631] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10520.809925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10531.299503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10531.312333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.331797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10532.326317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10533.818324] Lustre: server umount lustre-OST0000 complete [10542.579053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.584388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10542.592070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10542.592080] Lustre: Skipped 2 previous similar messages [10542.595398] Lustre: Skipped 2 previous similar messages [10552.288141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10552.459364] Lustre: server umount lustre-MDT0000 complete [10552.821927] LustreError: 325624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.848634] LustreError: 325624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10556.757698] LustreError: 322010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579186 with bad export cookie 5384799520838119833 [10556.771644] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10556.775543] LustreError: 322010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10557.315495] Lustre: server umount lustre-MDT0001 complete [10568.988952] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [10572.111527] Key type lgssc unregistered [10572.442965] LNet: 328306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10572.450528] LNetError: 328306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10572.469808] LNet: Removed LNI 192.168.204.141@tcp [10573.549280] Key type .llcrypt unregistered [10573.555348] Key type ._llcrypt unregistered [10603.317598] Key type ._llcrypt registered [10603.328457] Key type .llcrypt registered [10603.534319] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [10604.849486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10605.088735] alg: No test for adler32 (adler32-zlib) [10606.392658] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [10606.685548] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [10608.440199] Key type lgssc registered [10610.151953] Lustre: Echo OBD driver; http://www.lustre.org/ [10619.696485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10619.713454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.410529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10626.774893] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10631.650703] LustreError: 329724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.685899] LustreError: 329724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10636.773875] LustreError: 329725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.037600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.629280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10642.917331] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10651.472291] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10658.059545] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10662.156257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.168702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.208513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.221028] Lustre: Skipped 2 previous similar messages [10668.514703] LustreError: 330453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.536022] LustreError: 330453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10668.561714] Lustre: server umount lustre-MDT0000 complete [10672.943395] LustreError: 330460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579302 with bad export cookie 17219859647090581316 [10672.961502] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10672.969870] LustreError: 330460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10673.452705] Lustre: server umount lustre-MDT0001 complete [10689.248192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.593715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10689.650220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10689.890152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10694.363633] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10703.096081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10703.255445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10703.293859] Lustre: 332313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10703.690710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10708.620495] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10716.882977] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10722.951113] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10734.206763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.603272] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10734.617676] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10734.629546] LustreError: 334197:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10734.639517] LustreError: 334197:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10734.660072] LustreError: 334197:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10734.747620] Lustre: server umount lustre-OST0000 complete [10734.751336] LustreError: 334197:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10745.403978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10745.790374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10746.115319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10753.864449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10755.026508] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10764.135460] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10775.281766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.399139] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10775.515361] Lustre: lustre-OST0001: new disk, initializing [10775.518630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10775.529545] Lustre: 335743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10775.629085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10782.052879] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [10785.881456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10785.886976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10785.937474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10789.538750] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10796.537348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10796.537921] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10796.552182] LustreError: Skipped 1 previous similar message [10796.557048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10796.591702] Lustre: Skipped 2 previous similar messages [10798.362037] Lustre: server umount lustre-OST0001 complete [10801.653727] LustreError: 334664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.670485] LustreError: 334664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10805.228602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10805.236118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10805.245161] LustreError: Skipped 1 previous similar message [10805.271054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10805.303667] Lustre: Skipped 2 previous similar messages [10806.773717] LustreError: 334663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.808960] LustreError: 334663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10808.784215] Lustre: server umount lustre-OST0000 complete [10816.483751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10816.497051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10816.517671] Lustre: Skipped 1 previous similar message [10816.529704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10816.537553] Lustre: Skipped 2 previous similar messages [10827.233514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10827.247122] Lustre: Skipped 4 previous similar messages [10828.256202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10828.869500] Lustre: server umount lustre-MDT0000 complete [10831.337429] LustreError: 333103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10833.897229] LustreError: 333053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579463 with bad export cookie 17219859647090581869 [10833.903652] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.916955] LustreError: 333053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10834.482818] Lustre: server umount lustre-MDT0001 complete [10847.487070] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [10850.635466] Key type lgssc unregistered [10850.984088] LNet: 337707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10850.997205] LNetError: 337707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10852.077144] LNet: Removed LNI 192.168.204.141@tcp [10853.118851] Key type .llcrypt unregistered [10853.121245] Key type ._llcrypt unregistered [10932.104997] Key type ._llcrypt registered [10932.109967] Key type .llcrypt registered [10932.220537] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [10947.848348] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [10949.043880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10949.076324] alg: No test for adler32 (adler32-zlib) [10950.177938] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [10950.414234] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [10952.104225] Key type lgssc registered [10953.222934] Lustre: Echo OBD driver; http://www.lustre.org/ [11000.123760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11000.147396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11001.647983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11001.682837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11001.820280] Lustre: lustre-MDT0000: new disk, initializing [11001.945087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11001.964241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11007.463829] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11022.869823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.209975] Lustre: 342644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11023.377935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11023.391458] Lustre: Skipped 1 previous similar message [11023.566904] Lustre: lustre-MDT0001: new disk, initializing [11023.830357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11023.941365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11023.968801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11031.650892] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11043.100226] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11050.122576] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11058.437323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11058.801158] Lustre: lustre-OST0000: new disk, initializing [11058.804576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11058.817961] Lustre: 344614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11058.926461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11065.407346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11065.421852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11065.550436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11066.225774] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11076.962666] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11082.380475] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11082.627848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11087.309774] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11087.609628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11091.429643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11091.446265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.480429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11096.550403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11096.560142] Lustre: Skipped 2 previous similar messages [11101.670745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.690022] Lustre: Skipped 1 previous similar message [11104.225052] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11104.537684] Lustre: server umount lustre-OST0000 complete [11110.884690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11110.896824] LustreError: Skipped 1 previous similar message [11110.899131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.913428] Lustre: Skipped 1 previous similar message [11110.920476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11111.906703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.404579] Lustre: server umount lustre-MDT0000 complete [11122.146679] LustreError: 342656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.185863] LustreError: 342656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11123.024797] LustreError: 342635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579752 with bad export cookie 8281049633601382234 [11123.029054] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11123.045308] LustreError: 342635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11123.636589] Lustre: server umount lustre-MDT0001 complete [11132.498827] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:56:00 (1787579760) [11140.784921] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:56:09 (1787579769) [11151.187358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11151.456197] LustreError: 347369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11151.533099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11156.873575] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11156.962779] LustreError: 347370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11162.090178] LustreError: 347369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.752268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.075667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11172.404305] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11181.582474] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11188.376680] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11195.407158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11195.968247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11202.643175] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11210.592402] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11218.056059] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11219.898307] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11221.793794] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11223.663467] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11225.518782] Lustre: Failing over lustre-MDT0000 [11225.933417] Lustre: server umount lustre-MDT0000 complete [11226.599796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.602454] LustreError: 347365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.613306] Lustre: Skipped 2 previous similar messages [11226.631748] LustreError: 347365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11236.835508] LustreError: 347370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.861996] LustreError: 347370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11242.979104] Lustre: 340868:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579856/real 1787579856] req@00000000b584b485 x1874413045753984/t0(0) o400->MGC192.168.204.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787579872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11243.053170] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11246.493827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.202570] LustreError: 340864:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000029ee9cf0 x1874413045759104/t0(0) o250->MGC192.168.204.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11252.618357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11252.671900] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11253.616510] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11257.462357] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11257.849510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11257.879747] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11257.927609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11268.470248] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11270.798752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11274.726808] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11277.149850] Lustre: Failing over lustre-MDT0000 [11277.528495] Lustre: server umount lustre-MDT0000 complete [11278.310270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.313502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11278.315944] LustreError: 349864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.315954] LustreError: 349864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11278.328036] Lustre: Skipped 1 previous similar message [11294.689606] Lustre: 340867:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579908/real 1787579908] req@000000004e1c5c34 x1874413045775104/t0(0) o400->MGC192.168.204.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787579924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11294.721719] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11297.105858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.294584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11305.350896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11305.858793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11310.452378] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11310.593440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11310.609593] Lustre: Skipped 2 previous similar messages [11310.665949] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11310.740778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11322.514744] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11325.220098] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11336.192809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11336.208719] Lustre: Skipped 1 previous similar message [11336.215336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11336.216120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.243817] Lustre: Skipped 4 previous similar messages [11338.753780] Lustre: server umount lustre-MDT0000 complete [11341.282273] LustreError: 347370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.323346] LustreError: 347370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [11342.890182] LustreError: 347350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579972 with bad export cookie 8281049633601384943 [11342.892407] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.906979] LustreError: 347350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11343.301728] Lustre: server umount lustre-MDT0001 complete [11358.028132] Lustre: server umount lustre-OST0000 complete [11379.463503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.295253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11385.632117] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11395.682864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.564587] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11408.496940] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11415.673777] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11424.260405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11426.762543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11431.986772] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11440.675214] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11447.275924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11447.291459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.305256] Lustre: Skipped 2 previous similar messages [11447.311598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11448.968541] Lustre: server umount lustre-OST0000 complete [11459.239460] Lustre: server umount lustre-MDT0000 complete [11461.603753] LustreError: 354443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.641812] LustreError: 354443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11464.134100] LustreError: 355561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580094 with bad export cookie 8281049633601385685 [11464.138088] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11464.152818] LustreError: 355561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11474.575681] Lustre: server umount lustre-MDT0001 complete [11486.864763] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [11489.495572] Key type lgssc unregistered [11489.779724] LNet: 357262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11489.785958] LNetError: 357262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11489.801782] LNet: Removed LNI 192.168.204.141@tcp [11490.712736] Key type .llcrypt unregistered [11490.716275] Key type ._llcrypt unregistered [11517.632724] Key type ._llcrypt registered [11517.636455] Key type .llcrypt registered [11517.838475] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [11519.219785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11519.321824] alg: No test for adler32 (adler32-zlib) [11520.444679] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [11520.749256] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [11522.482342] Key type lgssc registered [11523.591735] Lustre: Echo OBD driver; http://www.lustre.org/ [11531.504827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11531.521347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11533.047276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.343928] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11542.499192] LustreError: 358783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.516565] LustreError: 358783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11547.185649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.665421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11552.348382] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11559.938257] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11566.661242] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11573.795986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11574.386914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11577.454969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11580.365915] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11587.838197] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11603.426447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11603.432726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11603.449650] Lustre: Skipped 1 previous similar message [11603.463203] Lustre: Skipped 2 previous similar messages [11608.547295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11608.562459] Lustre: Skipped 2 previous similar messages [11613.671829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11613.682479] Lustre: Skipped 2 previous similar messages [11617.250144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11617.549888] Lustre: server umount lustre-MDT0000 complete [11618.796719] LustreError: 361272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.807312] LustreError: 361272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11621.508598] LustreError: 358761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580251 with bad export cookie 552837564746142075 [11621.516301] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.517874] LustreError: 358761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11621.800871] Lustre: server umount lustre-MDT0001 complete [11636.229632] Lustre: server umount lustre-OST0000 complete [11650.677437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.268745] LustreError: 362516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.373059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11656.469255] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11656.675755] LustreError: 362517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.794339] LustreError: 362516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.268257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.751888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11671.678312] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11679.631440] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11685.563109] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11692.340501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11692.618078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11696.771595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11698.886126] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11706.955742] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11722.728930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.732473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11722.745580] Lustre: Skipped 2 previous similar messages [11722.755991] Lustre: Skipped 2 previous similar messages [11724.912571] Lustre: server umount lustre-MDT0000 complete [11727.848958] LustreError: 362513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.889619] LustreError: 362513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11729.503418] LustreError: 362497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580359 with bad export cookie 552837564746143188 [11729.516509] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11730.108356] Lustre: server umount lustre-MDT0001 complete [11745.106900] Lustre: server umount lustre-OST0000 complete [11763.542280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11763.863210] LustreError: 366432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.923253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11769.893186] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11780.913056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11785.678376] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11793.121763] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11798.689605] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11807.305917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.814502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11807.820963] Lustre: Skipped 1 previous similar message [11808.828539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11815.416146] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [11825.183721] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11833.881090] Lustre: server umount lustre-OST0000 complete [11841.515631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.518870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.528620] Lustre: Skipped 2 previous similar messages [11847.769879] Lustre: server umount lustre-MDT0000 complete [11851.453075] LustreError: 366413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580481 with bad export cookie 552837564746144210 [11851.465315] LustreError: 366413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11851.478687] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11862.146990] Lustre: server umount lustre-MDT0001 complete [11872.497882] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [11875.786753] Key type lgssc unregistered [11876.130324] LNet: 369968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11876.141703] LNetError: 369968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11876.154261] LNet: Removed LNI 192.168.204.141@tcp [11877.238894] Key type .llcrypt unregistered [11877.242104] Key type ._llcrypt unregistered [11904.806787] Key type ._llcrypt registered [11904.811403] Key type .llcrypt registered [11904.938786] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [11906.064696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11906.135574] alg: No test for adler32 (adler32-zlib) [11907.493991] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [11907.751565] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [11909.441470] Key type lgssc registered [11910.686538] Lustre: Echo OBD driver; http://www.lustre.org/ [11919.770886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11919.797639] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11919.811631] LustreError: 371410:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11919.838105] LustreError: 371410:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11919.851288] LustreError: 371410:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11919.858150] LustreError: 371410:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11919.881370] LustreError: 371410:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11927.988800] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11927.996620] LustreError: 371809:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11928.015352] LustreError: 371809:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11928.021808] LustreError: 371809:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11928.027479] LustreError: 371809:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11928.034206] LustreError: 371809:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11949.668366] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [11952.976755] Key type lgssc unregistered [11953.244690] LNet: 372815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11953.254338] LNetError: 372815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11953.276279] LNet: Removed LNI 192.168.204.141@tcp [11954.320818] Key type .llcrypt unregistered [11954.326293] Key type ._llcrypt unregistered [11975.104076] Key type ._llcrypt registered [11975.107154] Key type .llcrypt registered [11975.237345] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [11989.956186] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [11990.965380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11991.077129] alg: No test for adler32 (adler32-zlib) [11992.178805] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [11992.446539] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [11994.216389] Key type lgssc registered [11995.610984] Lustre: Echo OBD driver; http://www.lustre.org/ [12030.995231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12031.029288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.362562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12032.392983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12032.530664] Lustre: lustre-MDT0000: new disk, initializing [12032.635794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12032.707193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12037.217794] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12047.973747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.089391] Lustre: 375976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12048.170440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12048.179719] Lustre: Skipped 1 previous similar message [12048.260488] Lustre: lustre-MDT0001: new disk, initializing [12048.346411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12048.399622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12048.418570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12052.898831] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12063.861499] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12069.949720] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12077.978124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.209439] Lustre: lustre-OST0000: new disk, initializing [12078.211945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12078.215669] Lustre: 377943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12078.291420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12085.233590] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12087.389482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12087.409737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12087.527776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12095.614219] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12100.043144] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12100.280637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12105.723688] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12106.006651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12108.262318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12108.269442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.284412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12108.291890] Lustre: Skipped 1 previous similar message [12108.307201] Lustre: Skipped 1 previous similar message [12113.384887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12113.392352] Lustre: Skipped 1 previous similar message [12114.147857] Lustre: server umount lustre-OST0000 complete [12118.498660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12118.503129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12118.514343] Lustre: Skipped 1 previous similar message [12118.527410] Lustre: Skipped 1 previous similar message [12123.617434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12123.628654] Lustre: Skipped 1 previous similar message [12123.775833] Lustre: server umount lustre-MDT0000 complete [12127.570362] LustreError: 375966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580757 with bad export cookie 17404438148349227304 [12127.575171] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12127.586605] LustreError: 375966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12127.950829] Lustre: server umount lustre-MDT0001 complete [12136.096169] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12137.626487] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:12:46 (1787580766) [12143.981271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12144.437237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.538439] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12149.734979] LustreError: 380404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.758583] LustreError: 380404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12154.852682] LustreError: 380403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.610597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.866067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12162.170448] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12170.205611] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12176.225700] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12183.427521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12183.713142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12190.711399] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12198.109489] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12205.001158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12205.197669] Lustre: lustre-OST0001: new disk, initializing [12205.211851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12205.219690] Lustre: 383239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12205.313640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12210.808619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12210.814953] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12210.896297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12211.062984] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12220.135343] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12226.534769] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12226.535446] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.544255] LustreError: Skipped 2 previous similar messages [12226.549119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12226.553786] Lustre: Skipped 1 previous similar message [12230.948625] Lustre: server umount lustre-OST0001 complete [12231.656898] LustreError: 383248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.681354] LustreError: 383248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12236.771045] LustreError: 383248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.798858] LustreError: 383248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12241.905000] LustreError: 382226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.917642] LustreError: 382226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12247.015278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12247.034246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12247.040063] Lustre: Skipped 2 previous similar messages [12252.138795] LustreError: 382223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.154359] LustreError: 382223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12261.344347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12261.499896] Lustre: server umount lustre-OST0000 complete [12265.442454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12265.449948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12265.471072] Lustre: Skipped 1 previous similar message [12265.479613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12265.491087] Lustre: Skipped 5 previous similar messages [12279.264118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12279.406377] Lustre: server umount lustre-MDT0000 complete [12282.851259] LustreError: 380403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.874044] LustreError: 380403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12282.882071] LustreError: 380385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580912 with bad export cookie 17404438148349228186 [12282.890222] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12282.894274] LustreError: 380385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12297.184550] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12297.506129] Lustre: server umount lustre-MDT0001 complete [12308.281514] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [12311.110520] Key type lgssc unregistered [12311.368793] LNet: 385237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12311.378950] LNetError: 385237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12311.397376] LNet: Removed LNI 192.168.204.141@tcp [12312.308502] Key type .llcrypt unregistered [12312.311389] Key type ._llcrypt unregistered [12361.442958] Key type ._llcrypt registered [12361.447261] Key type .llcrypt registered [12361.543755] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [12363.173857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12363.204812] alg: No test for adler32 (adler32-zlib) [12364.431646] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [12364.688482] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [12366.408197] Key type lgssc registered [12367.650796] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.323817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.341074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.743900] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12376.767638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12376.970755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12383.235193] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12395.137731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12395.426341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12395.459518] Lustre: 388040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12395.809567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12402.215464] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12411.345745] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12417.385686] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12420.067798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12420.072834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12420.091467] Lustre: Skipped 1 previous similar message [12420.104996] Lustre: Skipped 1 previous similar message [12425.189147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12425.196580] Lustre: Skipped 1 previous similar message [12426.068595] Lustre: server umount lustre-MDT0000 complete [12430.067687] LustreError: 388032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581059 with bad export cookie 3916020013649670139 [12430.070461] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12430.082653] LustreError: 388032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12430.476410] Lustre: server umount lustre-MDT0001 complete [12440.483682] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:17:48 (1787581068) [12447.306539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.743821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12451.970288] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12452.842389] LustreError: 390553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.869036] LustreError: 390553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12457.957578] LustreError: 390554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.127079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12461.619496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.650146] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12474.699578] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12480.603570] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12487.303140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12487.517203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12487.770916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12493.725951] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12501.453742] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12510.412163] Lustre: Permanently deactivating lustre-OST0000 [12510.444183] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12517.965412] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12520.871798] LustreError: 393542:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12520.880250] LustreError: 393542:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12520.885617] LustreError: 393542:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12524.077504] LustreError: 393591:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12524.082815] LustreError: 393591:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12532.159626] Lustre: server umount lustre-OST0000 complete [12534.855813] LustreError: 393840:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12534.861336] LustreError: 393840:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12538.338576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12538.345001] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12538.362287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12541.409086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12541.413160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.530975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12546.541110] Lustre: Skipped 2 previous similar messages [12551.136935] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12551.302840] Lustre: server umount lustre-MDT0000 complete [12551.657221] LustreError: 390548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.675202] LustreError: 390548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12555.319252] LustreError: 390535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581185 with bad export cookie 3916020013649670720 [12555.329462] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12555.335452] LustreError: 390535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12566.143489] Lustre: server umount lustre-MDT0001 complete [12574.235708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.095984] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12582.477649] LustreError: 395000:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12588.175197] Lustre: 395339:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.141@tcp [12590.564280] Lustre: server umount MGS complete [12604.243309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.598317] LustreError: 396064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.631901] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12604.636338] Lustre: Skipped 1 previous similar message [12604.688191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12609.011140] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12610.034754] LustreError: 396065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12619.299920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12619.597049] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12624.369598] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12632.608926] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12638.935975] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12641.353955] Lustre: Permanently reactivating lustre-OST0000 [12641.392658] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12641.396777] Lustre: Skipped 2 previous similar messages [12668.384899] LustreError: 397833:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12676.812836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12677.251683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12677.258778] Lustre: Skipped 1 previous similar message [12684.589119] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12693.909319] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12703.205230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12703.214837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12703.239148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12704.225623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12707.221676] Lustre: server umount lustre-OST0000 complete [12714.468588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12714.478530] Lustre: Skipped 2 previous similar messages [12719.300481] Lustre: server umount lustre-MDT0000 complete [12719.587493] LustreError: 396792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.630304] LustreError: 396792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12723.660249] LustreError: 396044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581353 with bad export cookie 3916020013649671770 [12723.686673] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12724.073797] Lustre: server umount lustre-MDT0001 complete [12735.784807] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [12739.152772] Key type lgssc unregistered [12739.486952] LNet: 400433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12739.513114] LNetError: 400433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12739.551900] LNet: Removed LNI 192.168.204.141@tcp [12740.560136] Key type .llcrypt unregistered [12740.567327] Key type ._llcrypt unregistered [12763.137300] Key type ._llcrypt registered [12763.143374] Key type .llcrypt registered [12763.320446] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [12781.025167] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [12782.621488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12782.729441] alg: No test for adler32 (adler32-zlib) [12784.006141] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [12784.335208] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [12786.032283] Key type lgssc registered [12787.304676] Lustre: Echo OBD driver; http://www.lustre.org/ [12825.622929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12825.666607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12827.044732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12827.123966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12827.221746] Lustre: lustre-MDT0000: new disk, initializing [12827.329186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12827.375692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12832.438338] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12844.122619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12844.357396] Lustre: 403592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12844.451087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12844.458565] Lustre: Skipped 1 previous similar message [12844.577537] Lustre: lustre-MDT0001: new disk, initializing [12844.732867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12844.807486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12844.824405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12849.769182] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12860.757042] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12868.083112] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12877.537547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12878.017802] Lustre: lustre-OST0000: new disk, initializing [12878.023456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12878.032697] Lustre: 405561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12878.198537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12885.669060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12885.679391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12885.871133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12887.579066] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12898.762344] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12904.410778] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12904.624306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12910.376736] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12910.803076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12916.705375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12916.717028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.725505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12916.744946] Lustre: Skipped 1 previous similar message [12916.775700] Lustre: Skipped 1 previous similar message [12920.193154] Lustre: server umount lustre-OST0000 complete [12926.952798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12926.954820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12926.969585] LustreError: Skipped 1 previous similar message [12926.971310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12926.996264] Lustre: Skipped 1 previous similar message [12931.013334] Lustre: server umount lustre-MDT0000 complete [12934.978765] LustreError: 406903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581564 with bad export cookie 2280962174397036915 [12934.984290] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12934.991823] LustreError: 406903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12935.376113] Lustre: server umount lustre-MDT0001 complete [12944.253447] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12946.347695] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:26:14 (1787581574) [12953.551623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12953.883513] LustreError: 408024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12953.911207] LustreError: 408024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12953.972708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12959.202108] LustreError: 408025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12959.703385] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12963.306201] LustreError: 408024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12968.421369] LustreError: 408025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12969.408555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12969.775468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12974.336576] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [12983.353292] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12990.895993] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12999.391387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12999.932496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13006.854667] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13015.307589] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13017.613326] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13025.587786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13033.959571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13033.974576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13033.975382] Lustre: Skipped 1 previous similar message [13033.996194] Lustre: Skipped 2 previous similar messages [13037.603549] Lustre: server umount lustre-OST0000 complete [13044.195784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.212211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13044.231946] Lustre: Skipped 1 previous similar message [13048.398103] Lustre: server umount lustre-MDT0000 complete [13049.317391] LustreError: 408019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.337106] LustreError: 408019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13052.440313] LustreError: 408759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581682 with bad export cookie 2280962174397037790 [13052.444048] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13052.465200] LustreError: 408759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13052.738842] Lustre: server umount lustre-MDT0001 complete [13066.703602] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13071.089281] Key type lgssc unregistered [13071.531861] LNet: 411756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13071.545148] LNetError: 411756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13071.568422] LNet: Removed LNI 192.168.204.141@tcp [13073.027864] Key type .llcrypt unregistered [13073.029934] Key type ._llcrypt unregistered [13099.709780] Key type ._llcrypt registered [13099.712149] Key type .llcrypt registered [13099.808476] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13100.132617] Key type .llcrypt unregistered [13100.136419] Key type ._llcrypt unregistered [13120.222124] Key type ._llcrypt registered [13120.230166] Key type .llcrypt registered [13120.355242] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13121.858829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13121.893642] alg: No test for adler32 (adler32-zlib) [13123.139637] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13123.385880] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [13125.016419] Key type lgssc registered [13126.629952] Lustre: Echo OBD driver; http://www.lustre.org/ [13137.218632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13137.240799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13138.849544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13144.301606] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13148.137793] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13148.161201] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13153.249417] LustreError: 413977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13153.953267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13154.251735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13159.396712] LustreError: 413972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.424092] LustreError: 413972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13161.216951] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13163.515171] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13163.558789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13163.560822] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13168.617364] LustreError: 413977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.653982] LustreError: 413977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13171.396221] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.852503] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.878298] LustreError: 413976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13180.078973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13180.709927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13186.431603] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13196.259941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13196.277704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13196.289940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13196.304506] Lustre: Skipped 1 previous similar message [13196.320860] Lustre: Skipped 1 previous similar message [13201.379730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13201.391592] Lustre: Skipped 1 previous similar message [13201.681593] Lustre: server umount lustre-OST0000 complete [13206.497852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13206.505804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13206.509923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13206.526401] Lustre: Skipped 1 previous similar message [13206.547048] Lustre: Skipped 1 previous similar message [13211.618074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13211.629545] Lustre: Skipped 1 previous similar message [13212.826623] Lustre: server umount lustre-MDT0000 complete [13216.743987] LustreError: 413973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13216.773534] LustreError: 413973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13217.460650] LustreError: 413958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581847 with bad export cookie 10156620981150090561 [13217.466083] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13217.496895] LustreError: 413958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13217.795274] Lustre: server umount lustre-MDT0001 complete [13230.877982] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13234.011778] Key type lgssc unregistered [13234.372396] LNet: 417128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13234.383304] LNetError: 417128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13234.422388] LNet: Removed LNI 192.168.204.141@tcp [13235.280231] Key type .llcrypt unregistered [13235.283878] Key type ._llcrypt unregistered [13264.321340] Key type ._llcrypt registered [13264.325853] Key type .llcrypt registered [13264.479709] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13266.428887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13266.455180] alg: No test for adler32 (adler32-zlib) [13267.620940] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13268.030135] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [13269.792250] Key type lgssc registered [13271.149764] Lustre: Echo OBD driver; http://www.lustre.org/ [13280.849819] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13280.879660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13296.545115] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13301.664449] LustreError: 418520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.141@tcp: failed processing log, type 4: rc = -110 [13328.416212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13335.131816] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13343.875394] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13351.984619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13352.690269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13352.822584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13358.052353] LustreError: 418545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13358.073483] LustreError: 418545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13358.174346] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13363.183457] LustreError: 418546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13363.197405] LustreError: 418546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13367.982463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13368.291758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13368.302413] Lustre: Skipped 1 previous similar message [13368.487292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13373.687816] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13381.114685] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13386.443320] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13393.890161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13393.890376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13393.908178] LustreError: Skipped 1 previous similar message [13393.910610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13393.942883] Lustre: Skipped 1 previous similar message [13395.168771] Lustre: server umount lustre-OST0000 complete [13401.061758] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13401.077116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13401.100205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13401.110793] Lustre: Skipped 3 previous similar messages [13401.940677] Lustre: server umount lustre-MDT0000 complete [13404.649298] LustreError: 419562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13404.655907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13405.903459] LustreError: 418527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582035 with bad export cookie 18384923396111700503 [13405.907313] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13405.932811] LustreError: 418527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13406.282354] Lustre: server umount lustre-MDT0001 complete [13416.945298] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13420.759212] Key type lgssc unregistered [13421.050101] LNet: 422096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13421.056814] LNetError: 422096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13421.090902] LNet: Removed LNI 192.168.204.141@tcp [13422.501911] Key type .llcrypt unregistered [13422.503585] Key type ._llcrypt unregistered [13448.067616] Key type ._llcrypt registered [13448.070401] Key type .llcrypt registered [13448.193862] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13449.593444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13449.639514] alg: No test for adler32 (adler32-zlib) [13450.789643] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13451.033462] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [13452.688165] Key type lgssc registered [13453.789934] Lustre: Echo OBD driver; http://www.lustre.org/ [13461.880160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13461.906351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13463.395875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13467.570550] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13473.764182] LustreError: 423516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13473.779670] LustreError: 423516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13476.038409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13476.491225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13480.839169] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13489.571203] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13497.057978] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13505.449042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13505.886863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13510.012530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13512.469907] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13519.900548] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13526.403048] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) [13526.504870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13526.532780] Lustre: Skipped 1 previous similar message [13529.571528] Lustre: server umount lustre-MDT0000 complete [13531.505053] LustreError: 424765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13532.102417] LustreError: 423497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.41@tcp arrived at 1787582161 with bad export cookie 8423815362171290012 [13534.112044] LustreError: 423498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582163 with bad export cookie 8423815362171289123 [13534.129052] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13534.141402] LustreError: 423498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13536.738295] LustreError: 423511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13536.739485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13536.744167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13536.753097] LustreError: 423511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13536.775411] Lustre: Skipped 1 previous similar message [13536.780785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13536.787659] Lustre: Skipped 3 previous similar messages [13539.746227] LustreError: 423513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.748125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13539.766040] LustreError: 423513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13540.408612] Lustre: server umount lustre-OST0000 complete [13544.874449] LustreError: 423511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.418560] Lustre: server umount lustre-MDT0001 complete [13559.193738] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13563.060850] Key type lgssc unregistered [13563.432792] LNet: 427204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13563.449605] LNetError: 427204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13563.462681] LNet: Removed LNI 192.168.204.141@tcp [13564.525875] Key type .llcrypt unregistered [13564.530547] Key type ._llcrypt unregistered [13591.257027] Key type ._llcrypt registered [13591.260800] Key type .llcrypt registered [13591.381856] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13593.102269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13593.140909] alg: No test for adler32 (adler32-zlib) [13594.339721] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13594.598106] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [13596.305088] Key type lgssc registered [13597.597773] Lustre: Echo OBD driver; http://www.lustre.org/ [13605.842555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13605.864766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13607.382737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13612.494936] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13617.638372] LustreError: 428621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13617.659530] LustreError: 428621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13622.329324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13622.905484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13628.483694] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13637.091288] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13643.052947] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13650.602081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13651.070901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13653.133154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13659.047150] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13671.564107] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13678.049787] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13678.058253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13678.073498] Lustre: Skipped 1 previous similar message [13679.613325] LustreError: 428617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13679.702938] Lustre: server umount lustre-MDT0001 complete [13683.173193] LustreError: 428615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13683.207983] LustreError: 428615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13688.299300] LustreError: 428620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13688.334375] LustreError: 428620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13693.409561] LustreError: 429594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.413665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13693.454514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13693.460586] Lustre: Skipped 1 previous similar message [13696.225991] Lustre: server umount lustre-OST0000 complete [13701.618516] Lustre: server umount lustre-MDT0000 complete [13717.221509] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13720.904926] Key type lgssc unregistered [13721.280099] LNet: 432312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13721.287844] LNetError: 432312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13721.305701] LNet: Removed LNI 192.168.204.141@tcp [13722.394636] Key type .llcrypt unregistered [13722.397377] Key type ._llcrypt unregistered [13750.510952] Key type ._llcrypt registered [13750.513591] Key type .llcrypt registered [13750.662537] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [13766.176795] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13767.481959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13767.620892] alg: No test for adler32 (adler32-zlib) [13768.771415] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [13769.069932] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [13770.776297] Key type lgssc registered [13772.112274] Lustre: Echo OBD driver; http://www.lustre.org/ [13823.222579] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [13842.554921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13842.611496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13843.947308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13844.013497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13844.108365] Lustre: lustre-MDT0000: new disk, initializing [13844.177410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13844.194950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13849.663009] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13867.691795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13868.172601] Lustre: lustre-OST0000: new disk, initializing [13868.180626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13868.184431] Lustre: Skipped 1 previous similar message [13868.188557] Lustre: 437351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13868.248640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13875.662839] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13875.897260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13875.913024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13875.999497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13886.487491] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13901.562337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13901.696727] Lustre: 436330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13901.796614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13901.875410] Lustre: lustre-MDT0001: new disk, initializing [13902.063702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13902.099304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13902.109751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13907.486491] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [13917.799937] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13918.042320] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13923.147987] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13923.420468] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13929.884284] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.41@tcp (stopping) [13929.953356] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13929.968466] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13929.984509] Lustre: Skipped 1 previous similar message [13934.970664] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.41@tcp (stopping) [13934.980033] Lustre: Skipped 2 previous similar messages [13935.436555] Lustre: server umount lustre-MDT0001 complete [13945.321568] LustreError: 436344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.344050] LustreError: 436344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13950.457689] LustreError: 437604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13950.478756] LustreError: 437604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13955.564363] LustreError: 436344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13955.578846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13955.654254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13955.665785] Lustre: Skipped 2 previous similar messages [13960.675169] LustreError: 436343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13960.686424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13965.795407] LustreError: 436344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13965.798032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13968.354135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13968.555554] Lustre: server umount lustre-OST0000 complete [13979.093529] Lustre: server umount lustre-MDT0000 complete [13993.977396] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [13997.279624] Key type lgssc unregistered [13997.673414] LNet: 441109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13997.691206] LNetError: 441109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13997.709980] LNet: Removed LNI 192.168.204.141@tcp [13998.628801] Key type .llcrypt unregistered [13998.632790] Key type ._llcrypt unregistered [14040.591574] Key type ._llcrypt registered [14040.594877] Key type .llcrypt registered [14040.705929] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [14041.026284] Key type .llcrypt unregistered [14041.030322] Key type ._llcrypt unregistered [14060.626363] Key type ._llcrypt registered [14060.641398] Key type .llcrypt registered [14060.847952] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [14062.713562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14062.782939] alg: No test for adler32 (adler32-zlib) [14063.964991] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [14064.287343] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [14066.016167] Key type lgssc registered [14067.335524] Lustre: Echo OBD driver; http://www.lustre.org/ [14097.864040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14097.885599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14099.153801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14099.185051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14099.258475] Lustre: lustre-MDT0000: new disk, initializing [14099.327802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14099.344383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14104.714204] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14117.006295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14117.250439] Lustre: 444495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14117.340284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14117.350763] Lustre: Skipped 1 previous similar message [14117.530372] Lustre: lustre-MDT0001: new disk, initializing [14117.684673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14117.718313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14117.732784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14122.715550] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14134.156035] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14141.246448] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14149.384488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14149.738992] Lustre: lustre-OST0000: new disk, initializing [14149.742880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14149.748816] Lustre: 446464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14149.848905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14156.467295] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14159.484362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14159.514438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14159.623549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14166.839943] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14174.177864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14174.189936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14174.207289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14175.203824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14175.207514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14175.219964] Lustre: Skipped 1 previous similar message [14175.234224] Lustre: Skipped 2 previous similar messages [14178.799779] Lustre: server umount lustre-MDT0000 complete [14183.180463] LustreError: 444486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582813 with bad export cookie 7035683423319311990 [14183.183806] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14183.210092] LustreError: 444486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14183.886724] Lustre: server umount lustre-MDT0001 complete [14198.793674] Lustre: server umount lustre-OST0000 complete [14212.085799] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:47:20 (1787582840) [14221.356833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14221.947896] LustreError: 448494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14221.975859] LustreError: 448494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14222.072854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14227.429869] LustreError: 448495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14227.474411] LustreError: 448495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14228.241212] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14231.529418] LustreError: 448494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14236.646774] LustreError: 448495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14238.447030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14238.985328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14244.231564] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14254.238634] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14262.304748] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14283.946719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14284.162405] Lustre: Found index 0 for lustre-OST0000, updating log [14284.381481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14291.601648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14292.061167] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14300.495222] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14307.303233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14307.311366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14307.326715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14308.116473] Lustre: server umount lustre-OST0000 complete [14325.927488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14325.936341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14325.949289] Lustre: Skipped 1 previous similar message [14325.963387] Lustre: Skipped 1 previous similar message [14331.339832] Lustre: server umount lustre-MDT0000 complete [14331.361487] LustreError: 449930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14331.385040] LustreError: 449930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14335.056717] LustreError: 448474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582964 with bad export cookie 7035683423319313383 [14335.059627] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14335.070416] LustreError: 448474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14335.475560] Lustre: server umount lustre-MDT0001 complete [14344.144595] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:49:32 (1787582972) [14353.056365] LNet: 1 local NIs in recovery (showing 1): 192.168.204.141@tcp [14969.799259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14970.396303] LustreError: 454905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14970.489119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14975.992731] LustreError: 454906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14976.037365] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [14981.095114] LustreError: 454905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14985.943197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14986.370580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14991.595039] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15000.124682] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15006.170890] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15026.871040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15027.131746] Lustre: Found index 0 for lustre-OST0000, updating log [15027.431393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15031.971146] LNet: 1 local NIs in recovery (showing 1): 192.168.204.141@tcp [15035.017840] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15044.611861] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15048.102043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15048.122182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15053.564620] Lustre: server umount lustre-OST0000 complete [15059.366088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15059.374528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15059.384500] Lustre: Skipped 1 previous similar message [15059.402070] Lustre: Skipped 1 previous similar message [15060.459207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15060.466466] Lustre: Skipped 1 previous similar message [15062.502686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15062.507936] Lustre: Skipped 3 previous similar messages [15064.631962] Lustre: server umount lustre-MDT0000 complete [15065.575109] LustreError: 454902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15065.603511] LustreError: 454902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15068.170213] LustreError: 454885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583698 with bad export cookie 7035683423319314454 [15068.181123] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15068.191495] LustreError: 454885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15068.519145] Lustre: server umount lustre-MDT0001 complete [15091.360761] LNet: 1 local NIs in recovery (showing 1): 192.168.204.141@tcp [15100.721284] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [15112.557339] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [15153.164674] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:03:01 (1787583781) [15170.177696] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [15172.684967] Key type lgssc unregistered [15172.950803] LNet: 461536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15172.961990] LNetError: 461536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15287.654903] LNet: Removed LNI 192.168.204.141@tcp [15288.665900] Key type .llcrypt unregistered [15288.669254] Key type ._llcrypt unregistered [15305.400549] Key type ._llcrypt registered [15305.405442] Key type .llcrypt registered [15305.513414] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [15306.647523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15306.672883] alg: No test for adler32 (adler32-zlib) [15307.742657] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [15308.054674] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [15309.849326] Key type lgssc registered [15311.149938] Lustre: Echo OBD driver; http://www.lustre.org/ [15315.947673] LNet: Added LNI 192.168.204.141@tcp100 [8/256/0/180] [15316.875861] LNet: Added LNI 192.168.204.141@tcp101 [8/256/0/180] [15318.128557] LNet: Added LNI 192.168.204.141@tcp102 [8/256/0/180] [15319.136599] LNet: Added LNI 192.168.204.141@tcp103 [8/256/0/180] [15320.344600] LNet: Added LNI 192.168.204.141@tcp104 [8/256/0/180] [15321.545213] LNet: Added LNI 192.168.204.141@tcp105 [8/256/0/180] [15322.954070] LNet: Added LNI 192.168.204.141@tcp106 [8/256/0/180] [15324.085354] LNet: Added LNI 192.168.204.141@tcp107 [8/256/0/180] [15325.301907] LNet: Added LNI 192.168.204.141@tcp108 [8/256/0/180] [15326.392747] LNet: Added LNI 192.168.204.141@tcp109 [8/256/0/180] [15327.444843] LNet: Added LNI 192.168.204.141@tcp110 [8/256/0/180] [15328.637318] LNet: Added LNI 192.168.204.141@tcp111 [8/256/0/180] [15329.809807] LNet: Added LNI 192.168.204.141@tcp112 [8/256/0/180] [15331.084864] LNet: Added LNI 192.168.204.141@tcp113 [8/256/0/180] [15332.199156] LNet: Added LNI 192.168.204.141@tcp114 [8/256/0/180] [15333.622239] LNet: Added LNI 192.168.204.141@tcp115 [8/256/0/180] [15334.992601] LNet: Added LNI 192.168.204.141@tcp116 [8/256/0/180] [15336.402045] LNet: Added LNI 192.168.204.141@tcp117 [8/256/0/180] [15338.064088] LNet: Added LNI 192.168.204.141@tcp118 [8/256/0/180] [15339.480509] LNet: Added LNI 192.168.204.141@tcp119 [8/256/0/180] [15340.709684] LNet: Added LNI 192.168.204.141@tcp120 [8/256/0/180] [15342.314332] LNet: Added LNI 192.168.204.141@tcp121 [8/256/0/180] [15343.248917] LNet: Added LNI 192.168.204.141@tcp122 [8/256/0/180] [15344.143977] LNet: Added LNI 192.168.204.141@tcp123 [8/256/0/180] [15345.167869] LNet: Added LNI 192.168.204.141@tcp124 [8/256/0/180] [15346.605660] LNet: Added LNI 192.168.204.141@tcp125 [8/256/0/180] [15348.023703] LNet: Added LNI 192.168.204.141@tcp126 [8/256/0/180] [15349.064047] LNet: Added LNI 192.168.204.141@tcp127 [8/256/0/180] [15350.397268] LNet: Added LNI 192.168.204.141@tcp128 [8/256/0/180] [15351.243436] LNet: Added LNI 192.168.204.141@tcp129 [8/256/0/180] [15352.083731] LNet: Added LNI 192.168.204.141@tcp130 [8/256/0/180] [15353.022452] LNet: Added LNI 192.168.204.141@tcp131 [8/256/0/180] [15353.955389] LNet: Added LNI 192.168.204.141@tcp132 [8/256/0/180] [15355.389914] LNet: Added LNI 192.168.204.141@tcp133 [8/256/0/180] [15356.174980] LNet: Added LNI 192.168.204.141@tcp134 [8/256/0/180] [15357.237606] LNet: Added LNI 192.168.204.141@tcp135 [8/256/0/180] [15365.035437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15365.072899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15366.214535] Lustre: 464571:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.141@tcp: can't add NID 192.168.204.141@tcp130: rc = -75 [15366.275112] LNetError: 462031:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15366.285055] LNetError: Protocol error connecting to 192.168.204.141@tcp116 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15366.302603] LNetError: Skipped 5 previous similar messages [15366.343485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15366.365817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15366.433051] LNet: 8 local NIs in recovery (showing 8): 192.168.204.141@tcp121, 192.168.204.141@tcp122, 192.168.204.141@tcp124, 192.168.204.141@tcp126, 192.168.204.141@tcp127, 192.168.204.141@tcp129, 192.168.204.141@tcp130, 192.168.204.141@tcp123 [15366.438400] Lustre: lustre-MDT0000: new disk, initializing [15366.511689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15366.529635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15367.331038] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15367.351316] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [15367.362820] LNetError: Protocol error connecting to 192.168.204.141@tcp108 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15367.374175] LNetError: Skipped 3 previous similar messages [15369.505214] LNetError: 464600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15369.529263] LNetError: 464600:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15369.558337] LNetError: Protocol error connecting to 192.168.204.141@tcp121 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15369.583158] LNetError: Skipped 11 previous similar messages [15371.633828] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15373.604372] LNetError: 462029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15373.620082] LNetError: 462029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15373.628556] LNetError: Protocol error connecting to 192.168.204.141@tcp121 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15381.797288] LNetError: 464591:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15381.809178] LNetError: Protocol error connecting to 192.168.204.141@tcp121 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15382.486261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15382.734913] Lustre: 464588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15382.792803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15382.797415] Lustre: Skipped 1 previous similar message [15382.882234] Lustre: lustre-MDT0001: new disk, initializing [15382.920630] Lustre: 465390:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.141@tcp: can't add NID 192.168.204.141@tcp130: rc = -75 [15382.933843] Lustre: 465390:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15383.063239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15383.096204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15383.117242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15387.059588] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15389.988979] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15390.005018] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15390.012573] LNetError: Protocol error connecting to 192.168.204.141@tcp133 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15390.022288] LNetError: Skipped 9 previous similar messages [15392.307899] Lustre: 465822:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.141@tcp: can't add NID 192.168.204.141@tcp130: rc = -75 [15392.317596] Lustre: 465822:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15398.368960] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15404.359478] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15406.435402] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15406.449613] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15406.459333] LNetError: Protocol error connecting to 192.168.204.141@tcp125 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15406.472114] LNetError: Skipped 8 previous similar messages [15411.900387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15412.218876] Lustre: lustre-OST0000: new disk, initializing [15412.222482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15412.227715] Lustre: 466562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15412.339475] Lustre: 466567:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.141@tcp: can't add NID 192.168.204.141@tcp130: rc = -75 [15412.347787] Lustre: 466567:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15412.412657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15417.948090] Lustre: 466847:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.141@tcp: can't add NID 192.168.204.141@tcp130: rc = -75 [15417.978657] Lustre: 466847:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15418.078604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15418.112458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15418.271844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15419.482726] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15426.912331] LNet: 2 local NIs in recovery (showing 2): 192.168.204.141@tcp133, 192.168.204.141@tcp121 [15429.514932] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15438.818516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15438.819030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15438.826592] LustreError: Skipped 1 previous similar message [15438.829367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15438.848040] Lustre: Skipped 1 previous similar message [15438.862709] Lustre: Skipped 1 previous similar message [15440.226189] LNetError: 464591:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.141 [15440.249498] LNetError: 464591:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 28 previous similar messages [15440.272155] LNetError: Protocol error connecting to 192.168.204.141@tcp119 at host 192.168.204.141:988: is it running a compatible version of Lustre? [15440.279065] LNetError: Skipped 28 previous similar messages [15442.759156] Lustre: server umount lustre-OST0000 complete [15449.058565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15449.067160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15449.068390] Lustre: Skipped 1 previous similar message [15449.084522] Lustre: Skipped 1 previous similar message [15454.180811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15454.191477] Lustre: Skipped 1 previous similar message [15458.285301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15458.298705] Lustre: Skipped 1 previous similar message [15461.344312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15461.577853] Lustre: server umount lustre-MDT0000 complete [15465.530278] LustreError: 464579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584095 with bad export cookie 5922588621800453588 [15465.560462] LustreError: 464579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15465.563061] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15465.867299] Lustre: server umount lustre-MDT0001 complete [15479.202834] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [15483.534591] Key type lgssc unregistered [15484.190867] LNet: 468481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15484.197251] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15484.203536] LNet: Removed LNI 192.168.204.141@tcp [15485.216547] LNet: Removed LNI 192.168.204.141@tcp100 [15485.225317] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15485.248130] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15485.260943] LNet: Removed LNI 192.168.204.141@tcp101 [15486.304165] LNet: Removed LNI 192.168.204.141@tcp102 [15486.310157] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15486.326335] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15486.338945] LNet: Removed LNI 192.168.204.141@tcp103 [15487.392181] LNet: Removed LNI 192.168.204.141@tcp104 [15488.416630] LNet: Removed LNI 192.168.204.141@tcp105 [15488.422966] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15488.442984] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15488.466800] LNet: Removed LNI 192.168.204.141@tcp106 [15489.504334] LNet: Removed LNI 192.168.204.141@tcp107 [15490.528582] LNet: Removed LNI 192.168.204.141@tcp108 [15491.556502] LNet: Removed LNI 192.168.204.141@tcp109 [15492.576895] LNet: Removed LNI 192.168.204.141@tcp110 [15492.583856] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15492.602323] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15492.611681] LNet: Removed LNI 192.168.204.141@tcp111 [15493.665378] LNet: Removed LNI 192.168.204.141@tcp112 [15494.688490] LNet: Removed LNI 192.168.204.141@tcp113 [15495.712174] LNet: Removed LNI 192.168.204.141@tcp114 [15496.738295] LNet: Removed LNI 192.168.204.141@tcp115 [15497.760455] LNet: Removed LNI 192.168.204.141@tcp116 [15498.784544] LNet: Removed LNI 192.168.204.141@tcp117 [15499.808191] LNet: Removed LNI 192.168.204.141@tcp118 [15500.832894] LNet: Removed LNI 192.168.204.141@tcp119 [15500.835796] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15500.843572] LNetError: 468481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15500.850668] LNet: Removed LNI 192.168.204.141@tcp120 [15500.853883] LNet: Removed LNI 192.168.204.141@tcp121 [15501.920546] LNet: Removed LNI 192.168.204.141@tcp122 [15502.946737] LNet: Removed LNI 192.168.204.141@tcp123 [15503.968222] LNet: Removed LNI 192.168.204.141@tcp124 [15504.992986] LNet: Removed LNI 192.168.204.141@tcp125 [15506.016319] LNet: Removed LNI 192.168.204.141@tcp126 [15507.040188] LNet: Removed LNI 192.168.204.141@tcp127 [15508.064911] LNet: Removed LNI 192.168.204.141@tcp128 [15509.090694] LNet: Removed LNI 192.168.204.141@tcp129 [15510.112923] LNet: Removed LNI 192.168.204.141@tcp130 [15511.136319] LNet: Removed LNI 192.168.204.141@tcp131 [15512.160134] LNet: Removed LNI 192.168.204.141@tcp132 [15512.162452] LNet: Removed LNI 192.168.204.141@tcp133 [15513.184541] LNet: Removed LNI 192.168.204.141@tcp134 [15514.219084] LNet: Removed LNI 192.168.204.141@tcp135 [15514.927741] Key type .llcrypt unregistered [15514.929767] Key type ._llcrypt unregistered [15540.734748] Key type ._llcrypt registered [15540.736822] Key type .llcrypt registered [15540.844755] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [15557.126402] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [15558.585894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15558.665070] alg: No test for adler32 (adler32-zlib) [15559.693910] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [15560.009271] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [15561.704334] Key type lgssc registered [15563.082585] Lustre: Echo OBD driver; http://www.lustre.org/ [15602.247125] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:10:30 (1787584230) [15621.200476] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [15624.049973] Key type lgssc unregistered [15624.320889] LNet: 472572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15624.331647] LNetError: 472572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15624.343113] LNet: Removed LNI 192.168.204.141@tcp [15625.086815] Key type .llcrypt unregistered [15625.092714] Key type ._llcrypt unregistered [15643.632432] Key type ._llcrypt registered [15643.635459] Key type .llcrypt registered [15643.727597] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [15644.837628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15644.971070] alg: No test for adler32 (adler32-zlib) [15646.019602] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [15646.228121] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [15647.864377] Key type lgssc registered [15649.266205] Lustre: Echo OBD driver; http://www.lustre.org/ [15659.933152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15659.958046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15661.270308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15661.297890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15661.362579] Lustre: lustre-MDT0000: new disk, initializing [15661.529965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15661.546930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15667.954889] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15685.768433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15685.941782] Lustre: 473788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15686.036957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15686.046691] Lustre: Skipped 1 previous similar message [15686.246251] Lustre: lustre-MDT0001: new disk, initializing [15686.466457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15686.511675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15686.526232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15693.007415] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15706.142885] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15713.073448] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15721.752055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15721.991986] Lustre: lustre-OST0000: new disk, initializing [15722.000288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15722.004528] Lustre: 475758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15722.111366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15728.240798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15728.259958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15728.400226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15729.099510] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [15739.828893] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15744.328477] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing lnet_if_list [15750.409380] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15750.455479] LNet: Added LNI 192.168.204.141@tcp10 [8/256/0/180] [15756.637442] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15756.670976] LNet: Added LNI 192.168.204.141@tcp11 [8/256/0/180] [15761.460996] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15761.484592] LNet: Added LNI 192.168.204.141@tcp12 [8/256/0/180] [15766.118293] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15766.154620] LNet: Added LNI 192.168.204.141@tcp13 [8/256/0/180] [15771.348916] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15771.391417] LNet: Added LNI 192.168.204.141@tcp14 [8/256/0/180] [15776.380472] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15776.400705] LNet: Added LNI 192.168.204.141@tcp15 [8/256/0/180] [15781.572861] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15781.628717] LNet: Added LNI 192.168.204.141@tcp16 [8/256/0/180] [15786.339413] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15786.380055] LNet: Added LNI 192.168.204.141@tcp17 [8/256/0/180] [15791.687424] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15791.713956] LNet: Added LNI 192.168.204.141@tcp18 [8/256/0/180] [15797.338516] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15797.367927] LNet: Added LNI 192.168.204.141@tcp19 [8/256/0/180] [15804.147322] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15804.185922] LNet: Added LNI 192.168.204.141@tcp20 [8/256/0/180] [15810.779936] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15810.813447] LNet: Added LNI 192.168.204.141@tcp21 [8/256/0/180] [15817.294971] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15817.352792] LNet: Added LNI 192.168.204.141@tcp22 [8/256/0/180] [15823.641390] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15823.685382] LNet: Added LNI 192.168.204.141@tcp23 [8/256/0/180] [15830.050431] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15830.105199] LNet: Added LNI 192.168.204.141@tcp24 [8/256/0/180] [15837.251927] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15837.284234] LNet: Added LNI 192.168.204.141@tcp25 [8/256/0/180] [15844.663301] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15844.732811] LNet: Added LNI 192.168.204.141@tcp26 [8/256/0/180] [15851.546611] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15851.627482] LNet: Added LNI 192.168.204.141@tcp27 [8/256/0/180] [15859.840210] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15859.890456] LNet: Added LNI 192.168.204.141@tcp28 [8/256/0/180] [15868.180782] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15868.246690] LNet: Added LNI 192.168.204.141@tcp29 [8/256/0/180] [15875.251373] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15875.283535] LNet: Added LNI 192.168.204.141@tcp30 [8/256/0/180] [15881.668502] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15881.694117] LNet: Added LNI 192.168.204.141@tcp31 [8/256/0/180] [15888.745860] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15888.777651] LNet: Added LNI 192.168.204.141@tcp32 [8/256/0/180] [15896.848839] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15896.891261] LNet: Added LNI 192.168.204.141@tcp33 [8/256/0/180] [15904.168915] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15904.212798] LNet: Added LNI 192.168.204.141@tcp34 [8/256/0/180] [15912.133316] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15912.177608] LNet: Added LNI 192.168.204.141@tcp35 [8/256/0/180] [15921.541862] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15921.567706] LNet: Added LNI 192.168.204.141@tcp36 [8/256/0/180] [15932.080795] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15932.104529] LNet: Added LNI 192.168.204.141@tcp37 [8/256/0/180] [15941.526733] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15941.590642] LNet: Added LNI 192.168.204.141@tcp38 [8/256/0/180] [15952.198867] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15952.267152] LNet: Added LNI 192.168.204.141@tcp39 [8/256/0/180] [15960.813681] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15960.859092] LNet: Added LNI 192.168.204.141@tcp40 [8/256/0/180] [15970.194666] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15970.224911] LNet: Added LNI 192.168.204.141@tcp41 [8/256/0/180] [15980.888139] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15980.925943] LNet: Added LNI 192.168.204.141@tcp42 [8/256/0/180] [15990.183521] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15990.222222] LNet: Added LNI 192.168.204.141@tcp43 [8/256/0/180] [15999.922718] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15999.987371] LNet: Added LNI 192.168.204.141@tcp44 [8/256/0/180] [16009.442218] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16009.508815] LNet: Added LNI 192.168.204.141@tcp45 [8/256/0/180] [16020.267753] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16020.298173] LNet: Added LNI 192.168.204.141@tcp46 [8/256/0/180] [16030.531089] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16030.571406] LNet: Added LNI 192.168.204.141@tcp47 [8/256/0/180] [16041.015401] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16041.044748] LNet: Added LNI 192.168.204.141@tcp48 [8/256/0/180] [16049.985731] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16050.013586] LNet: Added LNI 192.168.204.141@tcp49 [8/256/0/180] [16059.680278] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16059.749570] LNet: Added LNI 192.168.204.141@tcp50 [8/256/0/180] [16080.865673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16080.884396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16080.905982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16081.891814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16081.896480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16081.922179] Lustre: Skipped 2 previous similar messages [16084.749849] Lustre: server umount lustre-MDT0000 complete [16089.747343] LustreError: 473779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584719 with bad export cookie 15243515594763877025 [16089.758797] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16089.771543] LustreError: 473779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16090.094938] Lustre: server umount lustre-MDT0001 complete [16108.513157] Lustre: 473062:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584722/real 1787584722] req@00000000ffd5b37d x1874417969978496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16108.560290] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16108.589039] Lustre: Skipped 1 previous similar message [16110.560860] Lustre: 473060:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584724/real 1787584724] req@00000000193681c4 x1874417969978624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16113.633595] Lustre: 473059:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584727/real 1787584727] req@00000000ef6c0850 x1874417969978752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16114.027943] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [16115.745691] Lustre: 473062:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584729/real 1787584729] req@00000000d8e11378 x1874417969979008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16120.928088] Lustre: 473062:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584734/real 1787584734] req@00000000bc2ae5d8 x1874417969979136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16124.588571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16134.240632] LustreError: 473058:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8e11378 x1874417969979904/t0(0) o250->MGC192.168.204.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16134.571580] LustreError: 475751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16134.595844] LustreError: 475751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16134.664073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16137.703534] LustreError: 475752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16137.707302] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16138.727968] LustreError: 475751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16139.729735] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16143.850276] LustreError: 475752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16148.442824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16148.830053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16153.779551] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16153.979727] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16162.652895] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16169.471668] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16190.946094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16190.959436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16190.985121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16195.041456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16195.056614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16195.057681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16195.084320] Lustre: Skipped 1 previous similar message [16196.569256] Lustre: server umount lustre-OST0000 complete [16205.284442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16205.286290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16205.300781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16205.318792] Lustre: Skipped 1 previous similar message [16206.615347] Lustre: server umount lustre-MDT0000 complete [16210.106692] LustreError: 473778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584839 with bad export cookie 15243515594763879664 [16210.109868] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16210.125226] LustreError: 473778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16210.471536] Lustre: server umount lustre-MDT0001 complete [16222.148869] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [16225.352830] Key type lgssc unregistered [16225.814374] LNet: 502812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16225.821728] LNetError: 502812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16225.828505] LNet: Removed LNI 192.168.204.141@tcp [16225.831938] LNet: Removed LNI 192.168.204.141@tcp10 [16225.834426] LNet: Removed LNI 192.168.204.141@tcp11 [16225.837522] LNet: Removed LNI 192.168.204.141@tcp12 [16225.842195] LNet: Removed LNI 192.168.204.141@tcp13 [16225.845878] LNet: Removed LNI 192.168.204.141@tcp14 [16225.849782] LNet: Removed LNI 192.168.204.141@tcp15 [16225.852608] LNet: Removed LNI 192.168.204.141@tcp16 [16225.855922] LNet: Removed LNI 192.168.204.141@tcp17 [16225.858844] LNet: Removed LNI 192.168.204.141@tcp18 [16225.864905] LNet: Removed LNI 192.168.204.141@tcp19 [16225.867899] LNet: Removed LNI 192.168.204.141@tcp20 [16225.871179] LNet: Removed LNI 192.168.204.141@tcp21 [16225.876457] LNet: Removed LNI 192.168.204.141@tcp22 [16225.880168] LNet: Removed LNI 192.168.204.141@tcp23 [16225.882368] LNet: Removed LNI 192.168.204.141@tcp24 [16225.884440] LNet: Removed LNI 192.168.204.141@tcp25 [16225.886865] LNet: Removed LNI 192.168.204.141@tcp26 [16225.889661] LNet: Removed LNI 192.168.204.141@tcp27 [16225.894488] LNet: Removed LNI 192.168.204.141@tcp28 [16225.899673] LNet: Removed LNI 192.168.204.141@tcp29 [16225.903241] LNet: Removed LNI 192.168.204.141@tcp30 [16225.908326] LNet: Removed LNI 192.168.204.141@tcp31 [16225.913632] LNet: Removed LNI 192.168.204.141@tcp32 [16225.918179] LNet: Removed LNI 192.168.204.141@tcp33 [16225.922372] LNet: Removed LNI 192.168.204.141@tcp34 [16225.937656] LNet: Removed LNI 192.168.204.141@tcp35 [16225.944659] LNet: Removed LNI 192.168.204.141@tcp36 [16225.953064] LNet: Removed LNI 192.168.204.141@tcp37 [16225.955297] LNet: Removed LNI 192.168.204.141@tcp38 [16225.965748] LNet: Removed LNI 192.168.204.141@tcp39 [16225.968792] LNet: Removed LNI 192.168.204.141@tcp40 [16225.971713] LNet: Removed LNI 192.168.204.141@tcp41 [16225.974368] LNet: Removed LNI 192.168.204.141@tcp42 [16225.976949] LNet: Removed LNI 192.168.204.141@tcp43 [16225.979769] LNet: Removed LNI 192.168.204.141@tcp44 [16225.981893] LNet: Removed LNI 192.168.204.141@tcp45 [16225.985369] LNet: Removed LNI 192.168.204.141@tcp46 [16225.988179] LNet: Removed LNI 192.168.204.141@tcp47 [16225.990386] LNet: Removed LNI 192.168.204.141@tcp48 [16225.992974] LNet: Removed LNI 192.168.204.141@tcp49 [16226.003853] LNet: Removed LNI 192.168.204.141@tcp50 [16226.946807] Key type .llcrypt unregistered [16226.950306] Key type ._llcrypt unregistered [16249.646474] Key type ._llcrypt registered [16249.650788] Key type .llcrypt registered [16249.800902] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [16266.326385] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [16267.431260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16267.633194] alg: No test for adler32 (adler32-zlib) [16268.689360] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [16268.904880] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [16270.593054] Key type lgssc registered [16272.130921] Lustre: Echo OBD driver; http://www.lustre.org/ [16313.256561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16313.277762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16314.567833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16314.617553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16314.730655] Lustre: lustre-MDT0000: new disk, initializing [16314.803312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16314.829303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16320.035305] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16332.236052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16332.323734] Lustre: 506015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16332.371355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16332.378206] Lustre: Skipped 1 previous similar message [16332.468577] Lustre: lustre-MDT0001: new disk, initializing [16332.569903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16332.634619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16332.641846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16337.668057] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16347.918261] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16353.929758] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16362.513931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16362.719831] Lustre: *** cfs_fail_loc=90f, val=0*** [16362.823400] Lustre: lustre-OST0000: new disk, initializing [16362.833459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16362.846181] Lustre: 508032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16362.972471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16371.264751] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16372.888343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16372.905151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16373.216041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16384.323209] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16393.703136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16393.734949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16393.772597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16393.779561] Lustre: Skipped 1 previous similar message [16398.846770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16398.865969] Lustre: Skipped 1 previous similar message [16399.290191] Lustre: server umount lustre-OST0000 complete [16403.941238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16403.956062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16403.974993] Lustre: Skipped 2 previous similar messages [16409.061583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16409.073437] Lustre: Skipped 2 previous similar messages [16409.596266] Lustre: server umount lustre-MDT0000 complete [16414.196548] LustreError: 506005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585044 with bad export cookie 6845214356031747028 [16414.204610] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16414.211651] LustreError: 506005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16414.711818] Lustre: server umount lustre-MDT0001 complete [16427.681664] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [16430.928592] Key type lgssc unregistered [16431.275903] LNet: 509944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16431.284218] LNetError: 509944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16431.301055] LNet: Removed LNI 192.168.204.141@tcp [16432.201326] Key type .llcrypt unregistered [16432.204731] Key type ._llcrypt unregistered [16454.211455] Key type ._llcrypt registered [16454.213962] Key type .llcrypt registered [16454.344823] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [16455.465880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16455.673529] alg: No test for adler32 (adler32-zlib) [16456.684942] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [16456.951204] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [16458.650155] Key type lgssc registered [16459.911275] Lustre: Echo OBD driver; http://www.lustre.org/ [16468.691583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16468.716192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16470.256363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16475.292119] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16480.739350] LustreError: 511217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16480.765185] LustreError: 511217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16485.865275] LustreError: 511218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16486.177266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16486.664898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16492.330934] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16501.565456] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16508.235108] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16518.619994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16519.354779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16529.038518] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16538.049883] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16553.142483] loop0: detected capacity change from 0 to 204800000 [16553.330561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16553.337786] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16553.347562] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16560.715279] loop0: detected capacity change from 0 to 204800000 [16561.038696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16561.073496] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16561.084958] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16566.885300] loop0: detected capacity change from 0 to 204800000 [16573.698363] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16573.876280] Lustre: fs15246-OST0000: new disk, initializing [16573.882822] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16573.890796] Lustre: 514813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16573.980026] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16579.958771] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16590.258846] loop1: detected capacity change from 0 to 204800000 [16596.441359] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16596.544921] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16596.591837] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16596.657364] Lustre: fs15246-MDT0000: new disk, initializing [16596.799699] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16596.821642] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16596.922207] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16597.986594] Lustre: 515937:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bd6f39f3 x1874418819188480/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787585243 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16601.717763] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16607.082514] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16607.101101] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16607.130823] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16631.279517] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16631.304366] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16632.816319] Lustre: server umount fs15246-MDT0000 complete [16636.404383] LustreError: 511214:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16641.507484] LustreError: 511214:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16644.570403] Lustre: server umount fs15246-OST0000 complete [16658.915773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16658.932252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16658.948725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16660.961405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16660.979911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16661.000211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16663.686765] Lustre: server umount lustre-OST0000 complete [16670.178390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16670.194361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16670.201130] Lustre: Skipped 1 previous similar message [16670.213479] Lustre: Skipped 3 previous similar messages [16674.486878] Lustre: server umount lustre-MDT0000 complete [16675.298132] LustreError: 514989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16675.324203] LustreError: 514989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16678.960972] LustreError: 511198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585308 with bad export cookie 1695590279365401044 [16678.967837] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16678.989424] LustreError: 511198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16679.440731] Lustre: server umount lustre-MDT0001 complete [16694.614964] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [16698.192305] Key type lgssc unregistered [16698.640187] LNet: 519039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16698.648126] LNetError: 519039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16698.674494] LNet: Removed LNI 192.168.204.141@tcp [16699.809640] Key type .llcrypt unregistered [16699.811816] Key type ._llcrypt unregistered [16719.792449] Key type ._llcrypt registered [16719.800478] Key type .llcrypt registered [16719.929586] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [16721.083206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16721.181809] alg: No test for adler32 (adler32-zlib) [16722.251412] Lustre: Lustre: Build Version: 2.17.57_80_ga77f2be [16722.465456] LNet: Added LNI 192.168.204.141@tcp [8/256/0/180] [16724.145832] Key type lgssc registered [16725.679326] Lustre: Echo OBD driver; http://www.lustre.org/ [16767.919557] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [16781.557730] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [16820.619517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16820.658897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16822.013933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16822.066072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16822.177994] Lustre: lustre-MDT0000: new disk, initializing [16822.259082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16822.277842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16826.411949] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16839.139322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16839.283069] Lustre: 523515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16839.347984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16839.353101] Lustre: Skipped 1 previous similar message [16839.476992] Lustre: lustre-MDT0001: new disk, initializing [16839.620774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16839.693113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16839.713055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16844.621931] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16854.676193] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16860.818944] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16868.380297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16868.696658] Lustre: lustre-OST0000: new disk, initializing [16868.700906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16868.707811] Lustre: 525484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16868.806343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16876.619676] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug -1 all [16877.687773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16877.695319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16877.852374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16888.447389] Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16892.921939] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16893.139942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16897.733836] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16898.019084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16903.650994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16903.654265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16903.678395] LustreError: Skipped 1 previous similar message [16903.680533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16903.722112] Lustre: Skipped 1 previous similar message [16906.561963] Lustre: server umount lustre-OST0000 complete [16911.332619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16911.352440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16911.382138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16911.388603] Lustre: Skipped 1 previous similar message [16913.899481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16913.906069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16918.051129] Lustre: server umount lustre-MDT0000 complete [16922.976053] LustreError: 523506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585552 with bad export cookie 3281224625769611698 [16922.987441] LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16922.989879] LustreError: 523506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16923.474865] Lustre: server umount lustre-MDT0001 complete [16954.099430] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_hostid [16965.075736] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing load_modules_local [17002.111469] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16680 sec ========= 11:33:49 (1787585629) [17004.926401] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:33:52 (1787585632) === [17009.885292] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:33:58 (1787585638) === [17054.067161] Lustre: DEBUG MARKER: oleg441-server.virtnet: executing unload_modules_local [17056.873664] Key type lgssc unregistered [17057.127882] LNet: 532135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17057.136094] LNetError: 532135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17057.169372] LNet: Removed LNI 192.168.204.141@tcp [17058.161836] Key type .llcrypt unregistered [17058.170794] Key type ._llcrypt unregistered