[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 459384873 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.003265] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.012032] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.022131] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027433] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.032236] *** VALIDATE proc *** [ 0.034189] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043994] debug: unmapping init [mem 0xffffffffb7c59000-0xffffffffb7c60fff] [ 0.046135] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055265] rcu: Hierarchical SRCU implementation. [ 0.057256] smp: Bringing up secondary CPUs ... [ 0.058617] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.064249] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.149020] node 0 deferred pages initialised in 79ms [ 0.153037] devtmpfs: initialized [ 0.154361] x86/mm: Memory block size: 128MB [ 0.156974] gcov: version magic: 0x41383552 [ 0.157960] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161151] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163304] pinctrl core: initialized pinctrl subsystem [ 0.165162] [ 0.165633] ************************************************************* [ 0.167011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173011] ** ** [ 0.175011] ** This means that this kernel is built to expose internal ** [ 0.177013] ** IOMMU data structures, which may compromise security on ** [ 0.179012] ** your system. ** [ 0.182012] ** ** [ 0.184010] ** If you see this message and you are not debugging the ** [ 0.186012] ** kernel, report this immediately to your vendor! ** [ 0.189033] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194010] ************************************************************* [ 0.197241] NET: Registered protocol family 16 [ 0.199523] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209003] cpuidle: using governor menu [ 0.210731] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213314] PCI: Using configuration type 1 for base access [ 0.216258] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228029] cryptd: max_cpu_qlen set to 1000 [ 0.230170] ACPI: Added _OSI(Module Device) [ 0.232014] ACPI: Added _OSI(Processor Device) [ 0.234028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241260] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247336] ACPI: Interpreter enabled [ 0.250074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252011] ACPI: Using IOAPIC for interrupt routing [ 0.255157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258346] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266929] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279223] acpiphp: Slot [2] registered [ 0.280116] acpiphp: Slot [5] registered [ 0.282112] acpiphp: Slot [6] registered [ 0.283123] acpiphp: Slot [7] registered [ 0.284124] acpiphp: Slot [8] registered [ 0.286075] acpiphp: Slot [9] registered [ 0.287091] acpiphp: Slot [10] registered [ 0.288103] acpiphp: Slot [3] registered [ 0.289067] acpiphp: Slot [4] registered [ 0.290130] acpiphp: Slot [11] registered [ 0.291066] acpiphp: Slot [12] registered [ 0.293078] acpiphp: Slot [13] registered [ 0.294035] acpiphp: Slot [14] registered [ 0.295066] acpiphp: Slot [15] registered [ 0.296060] acpiphp: Slot [16] registered [ 0.297075] acpiphp: Slot [17] registered [ 0.299107] acpiphp: Slot [18] registered [ 0.300097] acpiphp: Slot [19] registered [ 0.302105] acpiphp: Slot [20] registered [ 0.304130] acpiphp: Slot [21] registered [ 0.305074] acpiphp: Slot [22] registered [ 0.306063] acpiphp: Slot [23] registered [ 0.308085] acpiphp: Slot [24] registered [ 0.309099] acpiphp: Slot [25] registered [ 0.311113] acpiphp: Slot [26] registered [ 0.313174] acpiphp: Slot [27] registered [ 0.315123] acpiphp: Slot [28] registered [ 0.316074] acpiphp: Slot [29] registered [ 0.317080] acpiphp: Slot [30] registered [ 0.319099] acpiphp: Slot [31] registered [ 0.320118] PCI host bridge to bus 0000:00 [ 0.322019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333133] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340389] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349835] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367500] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377879] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414874] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454491] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514453] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.522019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.529013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.566309] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.587024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.618166] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.640023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.669680] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682692] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.689015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.712017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724167] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727254] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728211] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.729378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.732213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739031] iommu: Default domain type: Passthrough [ 0.740454] SCSI subsystem initialized [ 0.742123] ACPI: bus type USB registered [ 0.743132] usbcore: registered new interface driver usbfs [ 0.745077] usbcore: registered new interface driver hub [ 0.747090] usbcore: registered new device driver usb [ 0.749182] pps_core: LinuxPPS API ver. 1 registered [ 0.751013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754060] PTP clock support registered [ 0.758035] EDAC MC: Ver: 3.0.0 [ 0.760041] PCI: Using ACPI for IRQ routing [ 0.761503] NetLabel: Initializing [ 0.762009] NetLabel: domain hash size = 128 [ 0.764015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.766083] NetLabel: unlabeled traffic allowed by default [ 0.768301] vgaarb: loaded [ 0.770352] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.772018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.778523] clocksource: Switched to clocksource kvm-clock [ 0.889062] VFS: Disk quotas dquot_6.6.0 [ 0.891019] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.893839] *** VALIDATE ramfs *** [ 0.895080] *** VALIDATE hugetlbfs *** [ 0.897043] pnp: PnP ACPI init [ 0.899564] pnp: PnP ACPI: found 6 devices [ 0.922807] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.927026] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929667] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931289] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.933287] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.935961] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.938794] NET: Registered protocol family 2 [ 0.941359] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.946535] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.950412] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.955662] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.959851] TCP: Hash tables configured (established 65536 bind 65536) [ 0.963360] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.966807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969560] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971644] NET: Registered protocol family 1 [ 0.974744] RPC: Registered named UNIX socket transport module. [ 0.976575] RPC: Registered udp transport module. [ 0.978354] RPC: Registered tcp transport module. [ 0.980355] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983138] NET: Registered protocol family 44 [ 0.985048] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.987450] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989909] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992662] PCI: CLS 0 bytes, default 64 [ 0.995250] Unpacking initramfs... [ 2.393712] debug: unmapping init [mem 0xffff9a1ffcc54000-0xffff9a1ffffbffff] [ 2.397719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.400051] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.403235] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.925637] Initialise system trusted keyrings [ 2.927505] Key type blacklist registered [ 2.929773] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.938847] zbud: loaded [ 2.943837] *** VALIDATE nfs *** [ 2.945085] *** VALIDATE nfs4 *** [ 2.947022] pstore: using deflate compression [ 2.952308] Platform Keyring initialized [ 3.050542] NET: Registered protocol family 38 [ 3.051880] Key type asymmetric registered [ 3.053478] Asymmetric key parser 'x509' registered [ 3.054657] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.056810] io scheduler mq-deadline registered [ 3.058609] io scheduler kyber registered [ 3.060075] io scheduler bfq registered [ 3.061558] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.064097] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.066997] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.070236] ACPI: Power Button [PWRF] [ 3.076236] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.084104] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.094794] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.101339] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.115995] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.158652] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.187747] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.191919] Non-volatile memory driver v1.3 [ 3.193319] Linux agpgart interface v0.103 [ 3.219026] virtio_blk virtio1: [vda] 150408 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.223052] vda: detected capacity change from 0 to 77008896 [ 3.238498] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.240631] vdb: detected capacity change from 0 to 1073741824 [ 3.254900] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.257486] vdc: detected capacity change from 0 to 2621440000 [ 3.271627] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.274171] vdd: detected capacity change from 0 to 2621440000 [ 3.287137] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.290235] vde: detected capacity change from 0 to 4294967296 [ 3.306899] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.310291] vdf: detected capacity change from 0 to 4294967296 [ 3.315702] libphy: Fixed MDIO Bus: probed [ 3.321265] usbcore: registered new interface driver usbserial_generic [ 3.324303] usbserial: USB Serial support registered for generic [ 3.326238] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.330599] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.332475] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.334925] mousedev: PS/2 mouse device common for all mice [ 3.338053] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.338637] rtc_cmos 00:05: RTC can wake from S4 [ 3.348220] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.350103] rtc_cmos 00:05: registered as rtc0 [ 3.355423] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.355926] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.357802] intel_pstate: CPU model not supported [ 3.364463] hid: raw HID events driver (C) Jiri Kosina [ 3.367064] usbcore: registered new interface driver usbhid [ 3.369444] usbhid: USB HID core driver [ 3.371437] drop_monitor: Initializing network drop monitor service [ 3.373282] Initializing XFRM netlink socket [ 3.375667] NET: Registered protocol family 10 [ 3.378656] Segment Routing with IPv6 [ 3.379860] NET: Registered protocol family 17 [ 3.381855] mpls_gso: MPLS GSO support [ 3.387460] RAS: Correctable Errors collector initialized. [ 3.389644] AVX version of gcm_enc/dec engaged. [ 3.391108] AES CTR mode by8 optimization enabled [ 3.462930] sched_clock: Marking stable (3462911889, 0)->(4353030640, -890118751) [ 3.466765] registered taskstats version 1 [ 3.468770] Loading compiled-in X.509 certificates [ 3.470913] zswap: loaded using pool lzo/zbud [ 3.495453] Key type big_key registered [ 3.509990] Key type encrypted registered [ 3.511770] ima: No TPM chip found, activating TPM-bypass! [ 3.513894] ima: Allocated hash algorithm: sha1 [ 3.515646] ima: No architecture policies found [ 3.517548] evm: Initialising EVM extended attributes: [ 3.519692] evm: security.selinux [ 3.521412] evm: security.ima [ 3.522557] evm: security.capability [ 3.524010] evm: HMAC attrs: 0x1 [ 3.526866] rtc_cmos 00:05: setting system clock to 2026-09-22 17:22:35 UTC (1790097755) [ 3.533940] debug: unmapping init [mem 0xffffffffb8c03000-0xffffffffb8dfffff] [ 3.536594] debug: unmapping init [mem 0xffffffffb7982000-0xffffffffb7c58fff] [ 3.545194] Write protecting the kernel read-only data: 28672k [ 3.547970] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 3.551149] debug: unmapping init [mem 0xffffffffb6914000-0xffffffffb69fffff] [ 3.580177] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.590161] systemd[1]: Detected virtualization kvm. [ 3.592782] systemd[1]: Detected architecture x86-64. [ 3.594932] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.624342] systemd[1]: No hostname configured. [ 3.626268] systemd[1]: Set hostname to . [ 3.627849] random: systemd: uninitialized urandom read (16 bytes read) [ 3.630253] systemd[1]: Initializing machine ID from random generator. [ 3.679825] random: ln: uninitialized urandom read (6 bytes read) [ 3.770492] random: systemd: uninitialized urandom read (16 bytes read) [ 3.774749] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.784112] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.789720] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.426347] device-mapper: uevent: version 1.0.3 [ 4.428812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.091933] scsi host0: ata_piix [ 5.114432] scsi host1: ata_piix [ 5.117402] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.122562] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.140664] virtio_net virtio0 ens2: renamed from eth0 [ 5.151100] random: fast init done [ 8.970567] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.029336] random: crng init done [ 10.030764] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.475372] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.647445] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.896625] SELinux: Disabled at runtime. [ 11.956960] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.968320] systemd[1]: Detected virtualization kvm. [ 11.970381] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.478404] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.482739] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.489614] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.496443] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.500178] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.507826] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.513246] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Created slic[ 12.604638] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.983497] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.314847] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.336365] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.465683] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.480246] EDAC sbridge: Ver: 1.1.2 [ 15.184279] Key type dns_resolver registered [ 15.524260] NFS: Registering the id_resolver key type [ 15.526321] Key type id_resolver registered [ 15.527683] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg347-server login: [ 41.870450] libcfs: loading out-of-tree module taints kernel. [ 41.889706] Key type ._llcrypt registered [ 41.891357] Key type .llcrypt registered [ 41.938642] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 54.463086] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 56.081835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.094940] alg: No test for adler32 (adler32-zlib) [ 57.444722] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 58.355239] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 60.199157] Key type lgssc registered [ 61.947355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.795928] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.397802] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 131.038073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.064802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.252152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.298200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.379825] Lustre: lustre-MDT0000: new disk, initializing [ 132.479475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.506059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.408407] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.084851] hrtimer: interrupt took 6842025 ns [ 149.228961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.367870] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.419198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.424248] Lustre: Skipped 1 previous similar message [ 149.524825] Lustre: lustre-MDT0001: new disk, initializing [ 149.609422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.641233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.654505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.498409] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.487863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 165.350042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 165.601586] Lustre: lustre-OST0000: new disk, initializing [ 165.606565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 165.612851] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 165.688105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.020034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.030485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.101768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 171.781452] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.646321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.744993] Lustre: lustre-OST0001: new disk, initializing [ 184.753354] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 184.759615] Lustre: 9491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.865647] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.671489] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.610900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 194.620090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 194.657562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 201.793602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 208.119370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 214.429521] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing check_logdir /tmp/testlogs/ [ 219.412853] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing yml_node [ 223.660815] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 225.954937] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 228.413692] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 230.109070] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 22 13:26:21 EDT 2026 [ 246.198721] Lustre: DEBUG MARKER: only running tests: 33-75 [ 247.652606] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 248.884136] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 266.208088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.208390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.221331] Lustre: Skipped 2 previous similar messages [ 266.226532] Lustre: Skipped 3 previous similar messages [ 269.986707] Lustre: server umount lustre-MDT0000 complete [ 272.702908] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098024 with bad export cookie 3760195324664244826 [ 272.703269] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.713221] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 276.456834] LustreError: 6487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.457628] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.457970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 276.478832] LustreError: 6487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 276.506388] Lustre: Skipped 2 previous similar messages [ 277.472755] LustreError: 6488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.476744] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 277.503329] Lustre: Skipped 2 previous similar messages [ 282.592228] LustreError: 6489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.592387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.614841] LustreError: 6489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 287.199212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 287.385268] Lustre: server umount lustre-MDT0001 complete [ 291.061862] Lustre: server umount lustre-OST0000 complete [ 294.554104] Lustre: server umount lustre-OST0001 complete [ 301.048495] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 310.258381] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 340.992389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.199277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 341.222182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 341.278169] Lustre: lustre-MDT0000: new disk, initializing [ 341.338468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.353193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 344.477878] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 353.178734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.300383] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 353.368836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 353.373454] Lustre: Skipped 1 previous similar message [ 353.442318] Lustre: lustre-MDT0001: new disk, initializing [ 353.539629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 353.563228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 353.572657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 356.718709] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 364.318991] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 369.002611] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 374.390032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 374.616188] Lustre: lustre-OST0000: new disk, initializing [ 374.619615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 374.624789] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 374.692460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.522840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 376.539403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 376.658756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 379.587973] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 386.741533] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 389.912424] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 390.100317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 393.382561] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 393.548734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 397.284281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.288974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 397.307299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 397.314193] Lustre: Skipped 2 previous similar messages [ 401.098593] Lustre: server umount lustre-OST0000 complete [ 404.965829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 404.977943] LustreError: Skipped 1 previous similar message [ 404.981481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.989274] Lustre: Skipped 1 previous similar message [ 407.525495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 407.532239] Lustre: Skipped 2 previous similar messages [ 418.271156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 418.424653] Lustre: server umount lustre-MDT0000 complete [ 421.312512] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098173 with bad export cookie 3760195324664246359 [ 421.317885] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.320811] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 421.647381] Lustre: server umount lustre-MDT0001 complete [ 425.056317] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:29:36 (1790098176) [ 427.212581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 428.610151] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:29:40 (1790098180) [ 433.556268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.838303] LustreError: 19470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.904430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.865186] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 439.269261] LustreError: 19471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.896496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.098416] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 451.203394] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.815954] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 459.471429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.582062] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 469.753030] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.447281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.450781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.463244] LustreError: Skipped 1 previous similar message [ 489.465062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.479339] Lustre: Skipped 2 previous similar messages [ 489.512165] Lustre: Skipped 5 previous similar messages [ 491.284429] Lustre: server umount lustre-OST0000 complete [ 494.570088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.891411] Lustre: server umount lustre-MDT0000 complete [ 502.906850] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098254 with bad export cookie 3760195324664247241 [ 502.913806] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.920484] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.257739] Lustre: server umount lustre-MDT0001 complete [ 511.217448] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 513.137830] Key type lgssc unregistered [ 513.370525] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.378050] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.396077] LNet: Removed LNI 192.168.203.147@tcp [ 514.042140] Key type .llcrypt unregistered [ 514.045523] Key type ._llcrypt unregistered [ 529.955199] Key type ._llcrypt registered [ 529.958964] Key type .llcrypt registered [ 530.031782] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 530.558882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 530.617227] alg: No test for adler32 (adler32-zlib) [ 531.578747] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 531.742189] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 533.399345] Key type lgssc registered [ 534.172936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.835979] loop: module loaded [ 539.871248] loop0: detected capacity change from 0 to 204800000 [ 540.046770] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.052651] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.207419] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 544.903730] loop0: detected capacity change from 0 to 204800000 [ 545.029442] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.042788] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.057490] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 549.031931] loop0: detected capacity change from 0 to 204800000 [ 553.158088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 553.180672] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.332156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 554.371308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 554.430269] Lustre: lustre-MDT0000: new disk, initializing [ 554.483763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.501162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 557.254203] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 563.529385] loop1: detected capacity change from 0 to 204800000 [ 567.028938] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.198545] Lustre: lustre-OST07c6: new disk, initializing [ 567.201686] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 567.204795] Lustre: Skipped 1 previous similar message [ 567.210200] Lustre: 26197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 567.266711] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 568.156711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 568.161895] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 568.187422] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 570.777598] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 573.626832] Lustre: 26712:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 573.636314] Lustre: 26712:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 573.640400] Lustre: 26712:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 573.645144] Lustre: 26712:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 573.649234] Lustre: 26712:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 573.654802] Lustre: 26712:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 589.907426] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 589.911368] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 589.915236] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 589.919463] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 589.923573] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 589.927065] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 593.929609] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 593.937883] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 593.942596] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 593.945504] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.948914] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 593.952390] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.954905] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 593.960356] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.964036] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 593.968277] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.972042] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 593.975471] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 597.085374] Lustre: 26712:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 597.090790] Lustre: 26712:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 597.094793] Lustre: 26712:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 597.098844] Lustre: 26712:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 597.103688] Lustre: 26712:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 597.108263] Lustre: 26712:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 597.112741] Lustre: 26712:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 597.118344] Lustre: 26712:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 597.121870] Lustre: 26712:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 597.125811] Lustre: 26712:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 597.129252] Lustre: 26712:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 597.132893] Lustre: 26712:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.098225] Lustre: 25182:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 601.104928] Lustre: 25182:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 601.109756] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 601.114633] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.120365] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 601.127665] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.132505] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 601.137955] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.142918] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 601.148142] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 601.153226] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 601.158211] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 611.267913] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 611.272722] Lustre: 25183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 611.274915] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 611.277960] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 611.282050] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 611.286883] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 611.290338] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 611.294740] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 611.297921] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 611.300728] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 611.306646] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 611.310234] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 660.449626] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 660.450593] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.476068] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 665.571423] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 670.175121] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 670.266647] Lustre: server umount lustre-OST07c6 complete [ 676.447947] Lustre: server umount lustre-MDT0000 complete [ 686.533739] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 688.382459] Key type lgssc unregistered [ 688.610777] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.616640] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.630046] LNet: Removed LNI 192.168.203.147@tcp [ 689.358046] Key type .llcrypt unregistered [ 689.361378] Key type ._llcrypt unregistered [ 702.400750] Key type ._llcrypt registered [ 702.403297] Key type .llcrypt registered [ 702.468738] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 703.200794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 703.241290] alg: No test for adler32 (adler32-zlib) [ 704.201540] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 704.365563] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 705.999215] Key type lgssc registered [ 706.808953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.695323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.705971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.023503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.880603] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 722.404359] LustreError: 30018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.430668] LustreError: 30018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 722.732043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.950483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.603368] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 732.025624] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 736.066494] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 740.293357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 740.583035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.647245] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 749.517936] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 757.999913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 773.601612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.602890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.605832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.614436] LustreError: Skipped 1 previous similar message [ 773.625198] Lustre: Skipped 1 previous similar message [ 776.314994] Lustre: server umount lustre-OST0000 complete [ 779.233646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 779.240165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.249888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.255272] Lustre: Skipped 1 previous similar message [ 783.842534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.845502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.864549] Lustre: Skipped 1 previous similar message [ 788.960443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.057394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 793.067644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.072946] Lustre: Skipped 2 previous similar messages [ 793.214541] Lustre: server umount lustre-MDT0000 complete [ 795.762500] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098547 with bad export cookie 17413579793235508552 [ 795.774556] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 795.775602] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.066709] Lustre: server umount lustre-MDT0001 complete [ 804.695279] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 806.714392] Key type lgssc unregistered [ 806.931923] LNet: 33773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.937324] LNetError: 33773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.948329] LNet: Removed LNI 192.168.203.147@tcp [ 807.571718] Key type .llcrypt unregistered [ 807.574849] Key type ._llcrypt unregistered [ 821.219805] Key type ._llcrypt registered [ 821.224739] Key type .llcrypt registered [ 821.287798] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 821.960113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.078851] alg: No test for adler32 (adler32-zlib) [ 823.067070] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 823.255509] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 824.905390] Key type lgssc registered [ 825.863096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 831.512434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 831.535352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.823216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.685082] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 841.543072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.824805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 844.632880] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 849.577045] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 853.178290] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 857.303782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.620210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.854081] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 866.888974] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 877.112294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 887.273325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.281487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.285803] LustreError: Skipped 1 previous similar message [ 887.305428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.719568] Lustre: server umount lustre-OST0000 complete [ 892.387740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.394693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.395555] Lustre: Skipped 2 previous similar messages [ 892.398905] Lustre: Skipped 1 previous similar message [ 897.506876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.516641] Lustre: Skipped 2 previous similar messages [ 898.448034] Lustre: server umount lustre-MDT0000 complete [ 900.666711] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098652 with bad export cookie 12728878974174160437 [ 900.669872] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.676992] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 900.871772] Lustre: server umount lustre-MDT0001 complete [ 907.798830] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 909.710921] Key type lgssc unregistered [ 909.919557] LNet: 38589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.924288] LNetError: 38589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.939464] LNet: Removed LNI 192.168.203.147@tcp [ 910.474600] Key type .llcrypt unregistered [ 910.476738] Key type ._llcrypt unregistered [ 925.079482] Key type ._llcrypt registered [ 925.081526] Key type .llcrypt registered [ 925.149134] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 925.648570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.687085] alg: No test for adler32 (adler32-zlib) [ 926.620601] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 926.780511] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 928.400098] Key type lgssc registered [ 929.142670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.279273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.296464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.547846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.304467] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 943.613907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.872475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 946.569754] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 951.044906] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.681874] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.033720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.251357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.296363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 963.373567] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 967.898679] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 973.796712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.801344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.810687] Lustre: Skipped 1 previous similar message [ 973.824091] Lustre: Skipped 2 previous similar messages [ 974.394948] LustreError: 39996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.411057] LustreError: 39996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 974.424640] Lustre: server umount lustre-MDT0000 complete [ 976.399222] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098728 with bad export cookie 3646083460502286623 [ 976.401543] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.405599] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 978.913214] LustreError: 39997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.916348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.934347] Lustre: Skipped 1 previous similar message [ 978.939034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 981.922246] LustreError: 39997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.924537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 981.936709] LustreError: 39997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 981.942515] Lustre: Skipped 1 previous similar message [ 982.655401] Lustre: server umount lustre-MDT0001 complete [ 1010.143373] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1010.282658] Lustre: server umount lustre-OST0000 complete [ 1021.452868] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1023.357576] Key type lgssc unregistered [ 1023.600795] LNet: 43858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.604940] LNetError: 43858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.624704] LNet: Removed LNI 192.168.203.147@tcp [ 1024.211413] Key type .llcrypt unregistered [ 1024.214139] Key type ._llcrypt unregistered [ 1039.428614] Key type ._llcrypt registered [ 1039.431133] Key type .llcrypt registered [ 1039.498620] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1040.153595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.173418] alg: No test for adler32 (adler32-zlib) [ 1041.099966] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1041.263414] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1042.887235] Key type lgssc registered [ 1043.534533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1048.109825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1048.126642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.399439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.410547] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1057.987504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.227535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.175876] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1066.698610] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.861128] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1075.776555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.095274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.618070] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1086.613803] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1094.513099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1102.410353] Lustre: lustre-OST0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 1103.841098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.847868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.859780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.865642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.871678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.876972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.882483] Lustre: Skipped 1 previous similar message [ 1107.195581] Lustre: server umount lustre-OST0000 complete [ 1124.834397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1124.842500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.862118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.583957] Lustre: server umount lustre-MDT0000 complete [ 1129.441503] LustreError: 46006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.456537] LustreError: 46006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1130.935997] LustreError: 45248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098882 with bad export cookie 15825363781699879233 [ 1130.939780] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.946677] LustreError: 45248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1145.311305] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1145.507190] Lustre: server umount lustre-MDT0001 complete [ 1152.463849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1154.434756] Key type lgssc unregistered [ 1154.661645] LNet: 48994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.668190] LNetError: 48994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.684858] LNet: Removed LNI 192.168.203.147@tcp [ 1155.327234] Key type .llcrypt unregistered [ 1155.331280] Key type ._llcrypt unregistered [ 1171.564848] Key type ._llcrypt registered [ 1171.566635] Key type .llcrypt registered [ 1171.623950] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1172.039916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.082208] alg: No test for adler32 (adler32-zlib) [ 1173.094817] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1173.352705] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1175.015388] Key type lgssc registered [ 1175.842741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1181.157578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1181.170247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.491530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.944789] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1190.048359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.271183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.883620] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1197.053139] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1200.385082] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1205.065875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1205.315673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.360238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1210.723875] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1218.064515] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1223.714369] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1226.324892] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1237.473514] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-2213h43m08s [ 1238.413467] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1241.057320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.062036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.063857] Lustre: Skipped 2 previous similar messages [ 1241.072319] Lustre: Skipped 2 previous similar messages [ 1242.086870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.092197] Lustre: Skipped 3 previous similar messages [ 1245.640468] Lustre: server umount lustre-MDT0000 complete [ 1247.212312] LustreError: 50400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.225077] LustreError: 50400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1247.408609] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1248.226802] LustreError: 51325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.235684] LustreError: 51325:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1250.936499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.008452] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.147263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.181567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1252.202226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1252.956736] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1254.517972] Lustre: DEBUG MARKER: Wait for df (36909) ... [ 1255.244969] Lustre: DEBUG MARKER: done [ 1257.447880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.463080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.470722] Lustre: Skipped 2 previous similar messages [ 1262.564212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1262.569762] Lustre: Skipped 1 previous similar message [ 1262.636042] Lustre: server umount lustre-OST0000 complete [ 1267.682190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.682517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1267.686764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.691433] Lustre: Skipped 2 previous similar messages [ 1270.599182] Lustre: server umount lustre-MDT0000 complete [ 1272.786484] LustreError: 50388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099024 with bad export cookie 4528483247874804713 [ 1272.790773] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.792933] LustreError: 50388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1272.983524] Lustre: server umount lustre-MDT0001 complete [ 1278.267693] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1279.760031] Key type lgssc unregistered [ 1279.954551] LNet: 55321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.962495] LNetError: 55321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.979198] LNet: Removed LNI 192.168.203.147@tcp [ 1280.493221] Key type .llcrypt unregistered [ 1280.494929] Key type ._llcrypt unregistered [ 1302.420669] Key type ._llcrypt registered [ 1302.422496] Key type .llcrypt registered [ 1302.480415] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1302.960917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.016857] alg: No test for adler32 (adler32-zlib) [ 1303.956876] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1304.117949] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1305.735284] Key type lgssc registered [ 1306.459210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1310.990433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1311.002409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.144887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1312.158201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1312.249617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.637179] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1319.880201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.942750] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1319.960747] Lustre: 57523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.143026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.546692] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1326.858530] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1330.483673] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1334.582072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1334.688482] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1334.801994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.660318] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1340.500814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1343.249247] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.648654] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-2213h44m59s [ 1348.446657] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1352.438228] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1353.844451] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1353.866592] LustreError: 57528:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1374.282331] Lustre: lustre-MDT0000: Client 6de679e5-546d-4baa-9845-17d04ffd3554 (at 192.168.203.47@tcp) reconnecting [ 1393.911364] LustreError: 57528:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1394.982979] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1396.308903] Lustre: DEBUG MARKER: done [ 1399.778130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1399.785765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.796238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.834945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.836751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1401.846366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.181530] Lustre: server umount lustre-OST0000 complete [ 1407.457237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1407.461095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.470033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.473337] Lustre: Skipped 1 previous similar message [ 1412.577974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.581010] Lustre: Skipped 4 previous similar messages [ 1413.292823] Lustre: server umount lustre-MDT0000 complete [ 1415.231196] LustreError: 58263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099167 with bad export cookie 5323459784188068105 [ 1415.234475] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.240377] LustreError: 58263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1415.416392] Lustre: server umount lustre-MDT0001 complete [ 1420.643192] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1422.086828] Key type lgssc unregistered [ 1422.261368] LNet: 61843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.267974] LNetError: 61843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.287838] LNet: Removed LNI 192.168.203.147@tcp [ 1422.694393] Key type .llcrypt unregistered [ 1422.695898] Key type ._llcrypt unregistered [ 1444.985019] Key type ._llcrypt registered [ 1444.992763] Key type .llcrypt registered [ 1445.050735] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1445.472705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1445.484301] alg: No test for adler32 (adler32-zlib) [ 1446.405976] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1446.535519] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1448.143205] Key type lgssc registered [ 1448.938620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1451.150635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1451.164116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.334142] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1452.344208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1452.420481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.911050] Lustre: Failing over lustre-MDT0000 [ 1453.072102] Lustre: server umount lustre-MDT0000 complete [ 1459.213739] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:46:50 (1790099210) [ 1468.061750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.363348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.585784] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1483.402787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.516289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1483.595559] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1483.897693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.677911] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1498.351854] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1503.856620] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1509.625260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.833288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1510.024005] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.678602] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1515.600721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1520.438211] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.825843] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1530.849399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.851480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.858234] Lustre: Skipped 1 previous similar message [ 1532.445068] Lustre: server umount lustre-MDT0000 complete [ 1535.123718] LustreError: 64627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099287 with bad export cookie 16499016307344283179 [ 1535.126556] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.141089] LustreError: 64627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1535.355698] Lustre: server umount lustre-MDT0001 complete [ 1537.809991] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1541.182150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1547.963925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.327189] Lustre: 63547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099287/real 1790099287] req@0000000000685827 x1877055056678912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1551.366764] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.382368] Lustre: Skipped 1 previous similar message [ 1557.471969] Lustre: 63548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099293/real 1790099293] req@00000000441cc4c1 x1877055056679040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1560.938155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.014780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1561.953543] Lustre: 63549:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099297/real 1790099297] req@00000000a460cffa x1877055056679296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1564.561771] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1566.112248] Lustre: 63549:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099302/real 1790099302] req@0000000034856234 x1877055056679424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1567.209189] LustreError: 66484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.227783] LustreError: 66484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1570.281548] LustreError: 68404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.292948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1570.303969] LustreError: 68404:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1571.697974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.052716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1575.070282] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1577.458175] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1580.705380] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1585.253738] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1592.453411] Lustre: server umount lustre-MDT0000 complete [ 1592.802914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1592.809807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.813995] LustreError: 68404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.821830] Lustre: Skipped 2 previous similar messages [ 1592.839257] LustreError: 68404:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1595.637735] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099347 with bad export cookie 16499016307344285461 [ 1595.655607] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.658628] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1595.938576] Lustre: server umount lustre-MDT0001 complete [ 1600.467904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1607.112472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.215147] Lustre: 63549:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099349/real 1790099349] req@000000001685b71f x1877055056703488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1613.232438] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.472739] LustreError: 63546:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f336c09a x1877055056704640/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1621.672810] LustreError: 66483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.733793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.790621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1622.752825] Lustre: 63548:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790099358/real 1790099358] req@000000008fe6e01e x1877055056704000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790099374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1622.777357] Lustre: 63548:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1624.472836] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1629.153905] LustreError: 71086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.182731] LustreError: 71086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1630.146425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.491647] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1633.278948] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1638.509558] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1642.683263] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1648.097643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.099873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.115687] Lustre: Skipped 2 previous similar messages [ 1648.121210] Lustre: Skipped 3 previous similar messages [ 1649.951347] Lustre: server umount lustre-MDT0000 complete [ 1652.842903] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099404 with bad export cookie 16499016307344286840 [ 1652.846880] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.856491] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1653.097779] Lustre: server umount lustre-MDT0001 complete [ 1658.140358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1660.676788] Lustre: DEBUG MARKER: files compared the same [ 1671.310945] Lustre: server umount lustre-OST0000 complete [ 1683.841590] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1685.948390] Key type lgssc unregistered [ 1686.161793] LNet: 74356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.169863] LNetError: 74356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.187769] LNet: Removed LNI 192.168.203.147@tcp [ 1686.908310] Key type .llcrypt unregistered [ 1686.910272] Key type ._llcrypt unregistered [ 1704.433509] Key type ._llcrypt registered [ 1704.435219] Key type .llcrypt registered [ 1704.521363] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1705.215424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.231663] alg: No test for adler32 (adler32-zlib) [ 1706.188657] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1706.366372] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1707.999270] Key type lgssc registered [ 1708.941685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1714.427942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1714.439615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.697517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.534520] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug +malloc all [ 1724.304141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.509538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1727.519554] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug +malloc all [ 1733.573425] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1738.598769] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1744.681677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.062081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.142487] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug +malloc all [ 1755.766074] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1760.430636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1769.951754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.958372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.970930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.985192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.985480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.013970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.421448] Lustre: server umount lustre-OST0000 complete [ 1776.096038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.097037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.110405] Lustre: Skipped 1 previous similar message [ 1776.113801] Lustre: Skipped 1 previous similar message [ 1781.221251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.226219] Lustre: Skipped 1 previous similar message [ 1781.859201] Lustre: server umount lustre-MDT0000 complete [ 1785.235096] LustreError: 75751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099537 with bad export cookie 8036407443435214532 [ 1785.242894] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.251834] LustreError: 75751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.524733] Lustre: server umount lustre-MDT0001 complete [ 1795.197796] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1797.225360] Key type lgssc unregistered [ 1797.481789] LNet: 79299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.487860] LNetError: 79299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.500341] LNet: Removed LNI 192.168.203.147@tcp [ 1798.265653] Key type .llcrypt unregistered [ 1798.269300] Key type ._llcrypt unregistered [ 1816.539803] Key type ._llcrypt registered [ 1816.545161] Key type .llcrypt registered [ 1816.629905] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1817.402977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1817.419356] alg: No test for adler32 (adler32-zlib) [ 1818.406855] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1818.583640] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1820.207273] Key type lgssc registered [ 1821.020939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.086376] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.099499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.592624] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1847.711320] LustreError: 80683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 1874.335439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.885633] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1883.892944] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1889.332756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.428079] Lustre: *** cfs_fail_loc=706, val=0*** [ 1889.679634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.742671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1892.856718] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1895.913433] LustreError: 81767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.936973] LustreError: 81767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1899.071041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.347541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.634339] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1907.308764] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1910.761421] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1914.855412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.856397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.861222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.871117] Lustre: Skipped 1 previous similar message [ 1914.885286] Lustre: Skipped 1 previous similar message [ 1918.156403] Lustre: server umount lustre-OST0000 complete [ 1925.089635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.089821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.094769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.104049] Lustre: Skipped 1 previous similar message [ 1925.110076] Lustre: Skipped 1 previous similar message [ 1926.780721] Lustre: server umount lustre-MDT0000 complete [ 1929.077810] LustreError: 80689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099681 with bad export cookie 15364864866168607486 [ 1929.081784] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.091158] LustreError: 80689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1929.279440] Lustre: server umount lustre-MDT0001 complete [ 1935.721312] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1937.343211] Key type lgssc unregistered [ 1937.541080] LNet: 84294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.546325] LNetError: 84294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.564527] LNet: Removed LNI 192.168.203.147@tcp [ 1938.144276] Key type .llcrypt unregistered [ 1938.146181] Key type ._llcrypt unregistered [ 1953.760859] Key type ._llcrypt registered [ 1953.762565] Key type .llcrypt registered [ 1953.826894] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1954.363919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.383960] alg: No test for adler32 (adler32-zlib) [ 1955.335028] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1955.486540] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1957.095183] Key type lgssc registered [ 1957.836024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.770791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.793390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.484272] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1971.225504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.526857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.150125] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1976.802129] LustreError: 86391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.813302] LustreError: 86391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1979.029713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.236158] LustreError: 86390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.254143] LustreError: 86390:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1979.258983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.286296] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.299522] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1982.826905] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1986.531129] LustreError: 86391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.547124] LustreError: 86391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1988.336356] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.352349] Lustre: MGS: Client c5fb2664-6f54-427d-8579-ec7e971f4afd (at 0@lo) reconnecting [ 1988.372726] LustreError: 86405:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1988.583397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.644090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1991.469677] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2003.937572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.938467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.943791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.951464] Lustre: Skipped 1 previous similar message [ 2008.562625] Lustre: server umount lustre-OST0000 complete [ 2014.177465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.177966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.183201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.192571] Lustre: Skipped 1 previous similar message [ 2014.201528] Lustre: Skipped 1 previous similar message [ 2016.921836] Lustre: server umount lustre-MDT0000 complete [ 2019.272911] Lustre: server umount lustre-MDT0001 complete [ 2021.707316] Lustre: server umount MGS complete [ 2028.410670] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:56:20 (1790099780) [ 2046.272298] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 2051.808677] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2072.609708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.398868] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2081.897614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.051033] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2082.079888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2082.162210] Lustre: lustre-MDT0001: new disk, initializing [ 2082.211216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.237919] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2082.242296] Lustre: 93551:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2082.248875] LustreError: 93551:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2084.893866] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2092.081796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2092.242367] Lustre: lustre-OST0000: new disk, initializing [ 2092.245776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2092.254081] Lustre: 94317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2092.300669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.496459] Lustre: 94439:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cd9d2e1e x1877055590404864/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790099862 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2095.494050] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2100.744828] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.245561] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.258633] Lustre: MGS: Client 838919c7-af3e-4111-8816-f5896566a72b (at 0@lo) reconnecting [ 2104.290961] Lustre: 95362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2104.306460] LustreError: 93537:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2104.315362] LustreError: 93537:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2104.324474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2104.369792] Lustre: lustre-MDT0000: new disk, initializing [ 2104.449304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.461514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2104.504217] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2105.506156] Lustre: 95388:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008fe6e01e x1877055590411264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790099873 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2106.709574] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2114.602950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2114.611153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2114.678440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2140.641427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2140.642224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.645608] LustreError: Skipped 1 previous similar message [ 2140.650156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2140.657427] Lustre: Skipped 1 previous similar message [ 2140.671052] Lustre: Skipped 2 previous similar messages [ 2143.923826] Lustre: server umount lustre-OST0000 complete [ 2146.784370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2146.787683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.796284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.807221] Lustre: Skipped 1 previous similar message [ 2146.825478] Lustre: Skipped 3 previous similar messages [ 2151.904898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.911790] Lustre: Skipped 1 previous similar message [ 2152.221249] Lustre: server umount lustre-MDT0000 complete [ 2160.439393] Lustre: server umount lustre-MDT0001 complete [ 2162.620432] Lustre: server umount MGS complete [ 2169.683617] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:58:41 (1790099921) [ 2180.546382] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2182.228830] Key type lgssc unregistered [ 2182.399454] LNet: 97880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.404700] LNetError: 97880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.420388] LNet: Removed LNI 192.168.203.147@tcp [ 2182.909403] Key type .llcrypt unregistered [ 2182.911265] Key type ._llcrypt unregistered [ 2193.490657] Key type ._llcrypt registered [ 2193.493630] Key type .llcrypt registered [ 2193.554776] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2194.094381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.113255] alg: No test for adler32 (adler32-zlib) [ 2195.022675] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2195.153520] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2196.767302] Key type lgssc registered [ 2197.383735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.700595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.715284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.974584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.127325] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2209.969224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.192151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.770351] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2216.942058] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2220.057247] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.870981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.066808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.117603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2227.368424] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2231.330287] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2240.482434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.489388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.489508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.492919] Lustre: Skipped 1 previous similar message [ 2242.158771] Lustre: server umount lustre-OST0000 complete [ 2244.313509] LustreError: 101874:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000006ed7cefe) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2244.342575] Lustre: server umount lustre-MDT0000 complete [ 2245.600913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.601496] LustreError: 99825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.612568] Lustre: Skipped 1 previous similar message [ 2245.623039] LustreError: 99825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2246.424940] LustreError: 99833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099998 with bad export cookie 9589469541645114133 [ 2246.427346] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.434942] LustreError: 99833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2246.599388] Lustre: server umount lustre-MDT0001 complete [ 2251.985309] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2253.337538] Key type lgssc unregistered [ 2253.526440] LNet: 102626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.530956] LNetError: 102626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.544418] LNet: Removed LNI 192.168.203.147@tcp [ 2253.966691] Key type .llcrypt unregistered [ 2253.968468] Key type ._llcrypt unregistered [ 2262.696340] Key type ._llcrypt registered [ 2262.698337] Key type .llcrypt registered [ 2262.755144] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2263.299612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.308213] alg: No test for adler32 (adler32-zlib) [ 2264.226640] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2264.370719] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2265.992070] Key type lgssc registered [ 2266.745445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2270.402590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2270.411169] LustreError: 103719:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2270.411727] Lustre: Skipped 1 previous similar message [ 2270.411736] LustreError: 103717:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2270.427598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.431090] LustreError: 103719:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2270.431119] LustreError: 103719:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2270.431139] LustreError: 103719:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2271.719237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.045358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.216877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.399526] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2281.141278] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing lsmod [ 2282.640313] LustreError: 104921:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2282.654522] LustreError: 104923:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2282.658521] LustreError: 104921:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2282.667728] LustreError: 104921:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2282.670530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.673367] LustreError: 104921:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2282.916845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.439842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.446771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.449189] Lustre: Skipped 2 previous similar messages [ 2286.457215] Lustre: Skipped 2 previous similar messages [ 2290.614298] Lustre: server umount lustre-MDT0000 complete [ 2291.685201] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.696373] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2292.630104] LustreError: 104962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100044 with bad export cookie 913046170280613542 [ 2292.631732] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2292.634366] LustreError: 104962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2292.644246] LustreError: 104187:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e9baa6a6 x1877055914394624/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2292.811285] Lustre: server umount lustre-MDT0001 complete [ 2305.077502] Lustre: server umount lustre-OST0000 complete [ 2309.736842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.970338] LustreError: 105999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.009920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.970953] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2315.236791] LustreError: 105999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.252552] LustreError: 105999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2316.170759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.395409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2318.625866] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2322.724345] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.800230] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2329.578597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.756688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.118168] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2334.930063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2341.258186] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2350.560947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.564387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.576258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.585349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.589270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.889351] Lustre: server umount lustre-OST0000 complete [ 2361.827627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.893445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.903554] Lustre: Skipped 1 previous similar message [ 2366.945409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.955958] Lustre: Skipped 1 previous similar message [ 2374.116038] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2374.378889] Lustre: server umount lustre-MDT0000 complete [ 2376.163715] LustreError: 105999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.194763] LustreError: 105999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2379.672047] LustreError: 105980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100131 with bad export cookie 913046170280614221 [ 2379.683349] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.707817] LustreError: 105980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2380.040663] Lustre: server umount lustre-MDT0001 complete [ 2390.921902] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2393.385968] Key type lgssc unregistered [ 2393.604963] LNet: 109533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.614175] LNetError: 109533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.638126] LNet: Removed LNI 192.168.203.147@tcp [ 2394.334975] Key type .llcrypt unregistered [ 2394.338223] Key type ._llcrypt unregistered [ 2417.896340] Key type ._llcrypt registered [ 2417.898123] Key type .llcrypt registered [ 2418.021672] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2419.429869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.476134] alg: No test for adler32 (adler32-zlib) [ 2420.650613] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2420.986251] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2422.759298] Key type lgssc registered [ 2424.085704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2432.164038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2432.188040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.501552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.062446] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2443.746370] LustreError: 110964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.762270] LustreError: 110964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2444.274888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.577694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.433863] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2452.403223] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2456.245755] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2460.773593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.991864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.237942] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2470.825889] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.451965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2486.012485] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2488.373653] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2493.773638] Lustre: server umount lustre-OST0000 complete [ 2499.040605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.057358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.062529] Lustre: Skipped 1 previous similar message [ 2504.163497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.171147] Lustre: Skipped 1 previous similar message [ 2506.263813] Lustre: server umount lustre-MDT0000 complete [ 2508.921628] LustreError: 110946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100260 with bad export cookie 14742177370649099264 [ 2508.939190] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.661305] Lustre: server umount lustre-MDT0001 complete [ 2527.012665] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2529.117891] Key type lgssc unregistered [ 2529.319185] LNet: 114627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.326947] LNetError: 114627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.340929] LNet: Removed LNI 192.168.203.147@tcp [ 2529.944824] Key type .llcrypt unregistered [ 2529.946919] Key type ._llcrypt unregistered [ 2542.278322] Key type ._llcrypt registered [ 2542.281253] Key type .llcrypt registered [ 2542.355326] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2543.010933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.121083] alg: No test for adler32 (adler32-zlib) [ 2544.125492] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2544.301835] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2545.927216] Key type lgssc registered [ 2546.787709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.950718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.969638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.292217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.723761] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2562.683428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.957170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.998611] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2571.385475] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2575.138356] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2579.645727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.968572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.400427] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2589.692860] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2598.452504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2607.586321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2607.591523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.603080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.610651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.611014] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.611286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.470327] Lustre: server umount lustre-OST0000 complete [ 2618.850407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.854113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.856963] Lustre: Skipped 1 previous similar message [ 2618.872966] Lustre: Skipped 2 previous similar messages [ 2620.610272] Lustre: server umount lustre-MDT0000 complete [ 2623.535290] LustreError: 115823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100375 with bad export cookie 7330439188883757395 [ 2623.536508] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2623.546586] LustreError: 115823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2623.843205] Lustre: server umount lustre-MDT0001 complete [ 2632.148574] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2634.593167] Key type lgssc unregistered [ 2634.834762] LNet: 119393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.839266] LNetError: 119393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2634.858334] LNet: Removed LNI 192.168.203.147@tcp [ 2635.503101] Key type .llcrypt unregistered [ 2635.505644] Key type ._llcrypt unregistered [ 2656.930801] Key type ._llcrypt registered [ 2656.934554] Key type .llcrypt registered [ 2657.079516] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2657.894306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.930415] alg: No test for adler32 (adler32-zlib) [ 2658.883231] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2659.159911] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2660.919187] Key type lgssc registered [ 2661.786043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.602466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2668.628894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.052072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.201417] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2679.267037] LustreError: 120826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.276866] LustreError: 120826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2681.683220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.001764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.436415] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2698.727235] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2707.143275] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2715.646954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.118883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.243893] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2737.483562] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2739.741062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2748.988485] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2753.100705] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2753.107832] Lustre: Skipped 2 previous similar messages [ 2758.336751] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2758.341479] Lustre: Skipped 2 previous similar messages [ 2758.737280] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2758.787913] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2763.964448] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.47@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2763.986742] Lustre: Skipped 2 previous similar messages [ 2767.415917] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.47@tcp 192.168.0.[2,10]@tcp [ 2773.224928] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.47/24@tcp in log lustre-MDT0000 [ 2773.232117] Lustre: Skipped 2 previous similar messages [ 2775.689172] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.47/24@tcp [ 2775.695668] Lustre: Skipped 1 previous similar message [ 2785.766156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2785.775495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.787912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.794175] Lustre: Skipped 1 previous similar message [ 2790.758036] Lustre: server umount lustre-OST0000 complete [ 2795.880421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.883800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2795.889797] Lustre: Skipped 2 previous similar messages [ 2795.917551] Lustre: Skipped 1 previous similar message [ 2801.123940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2801.144550] Lustre: Skipped 1 previous similar message [ 2805.219041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2805.230603] Lustre: Skipped 1 previous similar message [ 2810.335940] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2810.337713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.369090] Lustre: Skipped 1 previous similar message [ 2810.638645] Lustre: server umount lustre-MDT0000 complete [ 2815.134318] LustreError: 120806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100567 with bad export cookie 5911964859216977465 [ 2815.148152] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.160072] LustreError: 120806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2815.752785] Lustre: server umount lustre-MDT0001 complete [ 2827.375958] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2830.402157] Key type lgssc unregistered [ 2830.740901] LNet: 125520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2830.749951] LNetError: 125520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2830.778165] LNet: Removed LNI 192.168.203.147@tcp [ 2831.520363] Key type .llcrypt unregistered [ 2831.522979] Key type ._llcrypt unregistered [ 2860.453750] Key type ._llcrypt registered [ 2860.456520] Key type .llcrypt registered [ 2860.578257] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2862.153081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2862.180957] alg: No test for adler32 (adler32-zlib) [ 2863.261736] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2863.596770] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2865.295703] Key type lgssc registered [ 2866.643596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2874.840307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2874.872080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.305486] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2876.327507] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.47/24@tcp [ 2876.351758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.770794] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2886.630506] LustreError: 127097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.651334] LustreError: 127097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2889.766552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.120546] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2890.164581] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.47/24@tcp [ 2890.213525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.848402] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2901.814768] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2907.528714] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2914.674787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.017864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.148851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2921.745590] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2929.486763] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2941.932406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.932970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.953667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.956893] Lustre: Skipped 1 previous similar message [ 2941.975766] Lustre: Skipped 1 previous similar message [ 2946.760951] Lustre: server umount lustre-OST0000 complete [ 2951.650255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2951.662526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.684852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.282154] LustreError: 127096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.291771] LustreError: 127096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2957.319913] Lustre: server umount lustre-MDT0000 complete [ 2961.081230] LustreError: 127077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100713 with bad export cookie 14634389428216611666 [ 2961.101706] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.389796] Lustre: server umount lustre-MDT0001 complete [ 2975.542841] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2979.518729] Key type lgssc unregistered [ 2979.803150] LNet: 130686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.811350] LNetError: 130686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.827856] LNet: Removed LNI 192.168.203.147@tcp [ 2980.992323] Key type .llcrypt unregistered [ 2980.994063] Key type ._llcrypt unregistered [ 3013.196137] Key type ._llcrypt registered [ 3013.201164] Key type .llcrypt registered [ 3013.305927] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 3027.760946] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3028.886183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3029.100680] alg: No test for adler32 (adler32-zlib) [ 3030.173235] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 3030.433790] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3032.112856] Key type lgssc registered [ 3033.223320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.655135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.684629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.955662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3070.986164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3071.116217] Lustre: lustre-MDT0000: new disk, initializing [ 3071.208288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.225496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3076.453884] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3087.973619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.083715] Lustre: 134098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3088.206331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3088.218246] Lustre: Skipped 1 previous similar message [ 3088.236450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3088.299965] Lustre: lustre-MDT0001: new disk, initializing [ 3088.417133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3088.457274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3088.465676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3092.948921] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3102.736911] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3109.356233] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3116.585226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.850969] Lustre: lustre-OST0000: new disk, initializing [ 3116.857308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3116.869193] Lustre: 136068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3116.981324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.795357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3122.814352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3122.973728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3124.494121] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3134.864671] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3139.458692] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3139.714373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3144.102913] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3144.299657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3148.768966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.781451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.789948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.794165] Lustre: Skipped 1 previous similar message [ 3152.918939] Lustre: server umount lustre-OST0000 complete [ 3159.009088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.017411] Lustre: Skipped 2 previous similar messages [ 3159.017600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.027735] Lustre: Skipped 1 previous similar message [ 3164.129315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.138304] Lustre: Skipped 1 previous similar message [ 3168.225976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.238232] Lustre: Skipped 1 previous similar message [ 3171.295151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3171.534899] Lustre: server umount lustre-MDT0000 complete [ 3175.454588] LustreError: 134092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100927 with bad export cookie 1589499324069735875 [ 3175.457468] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.472419] LustreError: 134092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3175.973751] Lustre: server umount lustre-MDT0001 complete [ 3185.270940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.927378] LustreError: 138285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.938820] LustreError: 138285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3185.977408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.840948] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3191.282194] LustreError: 138286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.367032] LustreError: 138285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.366695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.492275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3200.719950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.921594] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3214.673840] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3220.726830] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3228.488557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3228.812076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.482897] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3244.368348] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3249.463346] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3249.830311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3254.999932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3255.335303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3266.458620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.694210] Lustre: lustre-OST0001: new disk, initializing [ 3266.697368] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3266.704307] Lustre: 141659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3266.814599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.802329] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3274.877975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3274.887331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3274.965788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3281.875234] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3282.088463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3286.253602] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3286.447188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3292.155595] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90b254347000.ost_server_uuid 50 [ 3293.950430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90b254347000.ost_server_uuid in IDLE state after 0 sec [ 3305.983464] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.986456] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.013552] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3306.021358] Lustre: Skipped 1 previous similar message [ 3306.090761] LustreError: Skipped 2 previous similar messages [ 3308.128950] Lustre: server umount lustre-OST0001 complete [ 3311.086664] LustreError: 142088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.131765] LustreError: 142088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3313.120402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.138071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.199246] LustreError: 140106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.232544] LustreError: 140106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3318.767757] Lustre: server umount lustre-OST0000 complete [ 3323.872230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3323.880762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.899124] Lustre: Skipped 1 previous similar message [ 3330.020810] Lustre: server umount lustre-MDT0000 complete [ 3331.553424] LustreError: 138281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.574432] LustreError: 138281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3334.583193] LustreError: 141657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101086 with bad export cookie 1589499324069736757 [ 3334.589490] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.600148] LustreError: 141657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3335.029384] Lustre: server umount lustre-MDT0001 complete [ 3357.263996] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3360.493284] Key type lgssc unregistered [ 3360.750934] LNet: 144663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3360.759428] LNetError: 144663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3360.779834] LNet: Removed LNI 192.168.203.147@tcp [ 3361.608596] Key type .llcrypt unregistered [ 3361.611432] Key type ._llcrypt unregistered [ 3404.998750] Key type ._llcrypt registered [ 3405.002297] Key type .llcrypt registered [ 3405.097124] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 3424.233426] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3427.129640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3427.168324] alg: No test for adler32 (adler32-zlib) [ 3428.201869] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 3428.415471] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3430.111823] Key type lgssc registered [ 3431.858356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.672137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.718794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.987728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3475.032981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3475.096953] Lustre: lustre-MDT0000: new disk, initializing [ 3475.230524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.262227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3480.042848] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3492.699861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.878299] Lustre: 148836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3492.970343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3492.978554] Lustre: Skipped 1 previous similar message [ 3493.080728] Lustre: lustre-MDT0001: new disk, initializing [ 3493.197613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3493.251640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3493.267544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3498.443387] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3509.033238] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3515.658238] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3523.160426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.594692] Lustre: lustre-OST0000: new disk, initializing [ 3523.597785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3523.602174] Lustre: 150805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3523.680497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.713734] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3531.381827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3531.409809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3531.556056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3541.321532] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.524397] Lustre: Failing over lustre-OST0000 [ 3546.725272] Lustre: server umount lustre-OST0000 complete [ 3563.487275] Lustre: 147053:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101299/real 1790101299] req@000000006afdf17a x1877057134954624/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790101315 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3563.534561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.575097] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101299/real 1790101299] req@000000003acaf95f x1877057134954752/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790101315 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3564.630046] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3564.639782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.665226] Lustre: 147051:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101301/real 1790101301] req@00000000f09717cf x1877057134954880/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790101317 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3565.707259] Lustre: 147051:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3565.885280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.218608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.239126] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3568.801304] Lustre: 147054:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101304/real 1790101304] req@00000000fb031aab x1877057134956032/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790101320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3572.820094] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3573.347064] Lustre: 147054:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101309/real 1790101309] req@0000000058c1dc2d x1877057134957056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790101325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3573.377357] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3573.395987] Lustre: 147054:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3574.879350] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3574.899266] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3576.541638] Lustre: Failing over lustre-MDT0000 [ 3578.471528] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.502269] Lustre: Skipped 2 previous similar messages [ 3578.839838] Lustre: server umount lustre-MDT0000 complete [ 3583.972285] LustreError: 152483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.992707] LustreError: 152483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3585.069500] LustreError: 151184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.089231] LustreError: 148845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.110119] LustreError: 148845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3594.207401] Lustre: 147053:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790101330/real 1790101330] req@00000000cce28d04 x1877057134970496/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790101346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3594.212089] LustreError: 149948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.246645] Lustre: 147053:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3594.262230] LustreError: 149948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3594.272276] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.396070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.332141] LustreError: 148850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.343138] LustreError: 148850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3604.454194] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd116689e6b5a2f51 [ 3604.466261] Lustre: MGC192.168.203.147@tcp: Connection restored to 0@lo (at 0@lo) [ 3604.480399] Lustre: Skipped 1 previous similar message [ 3604.752215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.831920] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3605.558245] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3609.695091] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3610.090562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3610.108293] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3610.167345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3620.334854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.342837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.366101] Lustre: Skipped 1 previous similar message [ 3624.200275] Lustre: server umount lustre-OST0000 complete [ 3630.565862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.570923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3630.574078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.574086] Lustre: Skipped 1 previous similar message [ 3630.578212] Lustre: Skipped 1 previous similar message [ 3635.685526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.693836] Lustre: Skipped 2 previous similar messages [ 3640.800167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.810879] Lustre: Skipped 1 previous similar message [ 3642.335101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3642.565201] Lustre: server umount lustre-MDT0000 complete [ 3645.921229] LustreError: 148849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.954024] LustreError: 148849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3646.470245] LustreError: 148830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101398 with bad export cookie 15066344633029766993 [ 3646.477283] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.489237] LustreError: 148830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3646.831835] Lustre: server umount lustre-MDT0001 complete [ 3658.630261] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3661.888985] Key type lgssc unregistered [ 3662.199908] LNet: 154470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3662.206598] LNetError: 154470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3662.227909] LNet: Removed LNI 192.168.203.147@tcp [ 3663.197962] Key type .llcrypt unregistered [ 3663.200486] Key type ._llcrypt unregistered [ 3703.281183] Key type ._llcrypt registered [ 3703.283785] Key type .llcrypt registered [ 3703.493411] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3704.890920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3705.091067] alg: No test for adler32 (adler32-zlib) [ 3706.337615] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 3706.703182] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3708.423196] Key type lgssc registered [ 3709.678888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3718.016751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3718.045202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.547164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.768948] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3729.906265] LustreError: 156005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.926682] LustreError: 156005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3733.132859] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3735.008502] LustreError: 156006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.438277] LustreError: 156034:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3737.451500] LustreError: 156034:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3737.465699] LustreError: 156034:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3737.878965] Lustre: server umount lustre-MDT0000 complete [ 3755.921213] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 3767.477269] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3805.847404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.077422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3806.127257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3806.227484] Lustre: lustre-MDT0000: new disk, initializing [ 3806.315312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.329988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3811.150804] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3821.378342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.505912] Lustre: 159339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3821.587837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3821.588416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3821.595945] Lustre: Skipped 1 previous similar message [ 3821.664560] Lustre: lustre-MDT0001: new disk, initializing [ 3821.756563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.811296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3821.822115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3825.448810] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3834.147623] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3839.362902] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3846.085822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.338667] Lustre: lustre-OST0000: new disk, initializing [ 3846.343694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3846.349328] Lustre: 161306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3846.410045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.879305] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3855.432555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3855.444157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3855.576411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3860.992672] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3865.454330] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3867.184151] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:26:58 (1790101618) [ 4137.439574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.441205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.471340] Lustre: Skipped 1 previous similar message [ 4137.485961] Lustre: Skipped 2 previous similar messages [ 4141.957430] Lustre: server umount lustre-MDT0000 complete [ 4142.561145] LustreError: 159346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.583492] LustreError: 159346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4145.894348] LustreError: 161305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790101897 with bad export cookie 4650817379430354579 [ 4145.900950] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.908441] LustreError: 161305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4147.679849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.681553] LustreError: 159347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.692887] Lustre: Skipped 1 previous similar message [ 4147.693666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4150.687925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4152.433925] Lustre: server umount lustre-MDT0001 complete [ 4162.469095] Lustre: server umount lustre-OST0000 complete [ 4172.056428] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 4182.857947] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4217.971332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.304748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4218.356355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4218.489416] Lustre: lustre-MDT0000: new disk, initializing [ 4218.631716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.677067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4223.660102] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4233.847083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.950564] Lustre: 165981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4234.004669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4234.012668] Lustre: Skipped 1 previous similar message [ 4234.104527] Lustre: lustre-MDT0001: new disk, initializing [ 4234.198783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4234.262473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4234.283899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4238.692369] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4248.337865] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4254.554741] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4262.095114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4262.530745] Lustre: lustre-OST0000: new disk, initializing [ 4262.539130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4262.549540] Lustre: 167948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4262.665640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.631253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4268.639843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4268.802927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4269.350498] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4279.297654] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4284.171439] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4284.566635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4291.350084] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4291.637927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4294.627107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.632513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.643929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.656301] Lustre: Skipped 1 previous similar message [ 4300.333594] Lustre: server umount lustre-OST0000 complete [ 4304.866115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.871522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.888054] Lustre: Skipped 1 previous similar message [ 4304.906937] Lustre: Skipped 4 previous similar messages [ 4310.094869] Lustre: server umount lustre-MDT0000 complete [ 4314.443951] LustreError: 165972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102066 with bad export cookie 4650817379430513409 [ 4314.455158] LustreError: 165972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4314.455833] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4315.057703] Lustre: server umount lustre-MDT0001 complete [ 4324.345982] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:34:35 (1790102075) [ 4341.769966] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 4351.800706] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4389.938891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.276147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4390.329532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4390.461502] Lustre: lustre-MDT0000: new disk, initializing [ 4390.602244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.629395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4396.141344] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4407.148133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.270718] Lustre: 172704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4407.627977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4412.247934] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4422.020873] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4427.994543] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4435.355887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.731109] Lustre: lustre-OST0000: new disk, initializing [ 4435.734116] Lustre: Skipped 1 previous similar message [ 4435.738933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4435.743974] Lustre: Skipped 2 previous similar messages [ 4435.748793] Lustre: 174682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4441.171153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4441.185728] Lustre: Skipped 1 previous similar message [ 4441.195220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4441.330461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4442.962563] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4453.831824] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4462.057795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.059949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4462.064723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4462.069810] Lustre: Skipped 1 previous similar message [ 4462.094258] Lustre: Skipped 3 previous similar messages [ 4466.559512] Lustre: server umount lustre-OST0000 complete [ 4472.288245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.131474] Lustre: server umount lustre-MDT0000 complete [ 4477.415454] LustreError: 172717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.445828] LustreError: 172717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4480.984823] LustreError: 172698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102232 with bad export cookie 4650817379430514291 [ 4481.000418] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4481.005635] LustreError: 172698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4481.047274] LustreError: 155308:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 4481.461216] Lustre: server umount lustre-MDT0001 complete [ 4491.331533] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:37:22 (1790102242) [ 4507.541901] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 4520.246460] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4559.079825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.402346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4559.436757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4559.489404] Lustre: lustre-MDT0000: new disk, initializing [ 4559.547965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.553805] Lustre: Skipped 2 previous similar messages [ 4559.570910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4564.721233] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4578.882737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.054502] Lustre: 179008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4579.366830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4584.569932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4595.048935] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4601.525605] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4608.232139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4608.496520] Lustre: 180985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4613.719094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4613.918383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4615.418994] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4624.997858] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4634.594246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4634.599412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4634.600933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4634.600939] Lustre: Skipped 2 previous similar messages [ 4634.617490] Lustre: Skipped 2 previous similar messages [ 4638.943230] Lustre: server umount lustre-OST0000 complete [ 4649.485708] Lustre: server umount lustre-MDT0000 complete [ 4649.953248] LustreError: 180201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.977355] LustreError: 180201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4653.668460] LustreError: 178999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102405 with bad export cookie 4650817379430515376 [ 4653.671752] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.686515] LustreError: 178999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4654.093302] Lustre: server umount lustre-MDT0001 complete [ 4664.786665] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4667.294190] Key type lgssc unregistered [ 4667.628919] LNet: 182841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4667.636090] LNetError: 182841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4667.655693] LNet: Removed LNI 192.168.203.147@tcp [ 4668.508596] Key type .llcrypt unregistered [ 4668.512729] Key type ._llcrypt unregistered [ 4693.766408] Key type ._llcrypt registered [ 4693.770983] Key type .llcrypt registered [ 4693.892231] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4695.157689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4695.190948] alg: No test for adler32 (adler32-zlib) [ 4696.400088] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 4696.688453] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4698.447202] Key type lgssc registered [ 4700.150816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4708.393143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4708.413762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.895740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.279763] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4719.079301] LustreError: 184273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.101967] LustreError: 184273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4724.211023] LustreError: 184274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.159503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.573407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4729.722357] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4737.919108] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4744.116877] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4750.566250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.943371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.085423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4757.827978] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4766.174320] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4791.266969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4791.274651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.280365] LustreError: Skipped 1 previous similar message [ 4791.285422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4791.298625] Lustre: Skipped 1 previous similar message [ 4791.315793] Lustre: Skipped 1 previous similar message [ 4793.308829] Lustre: server umount lustre-OST0000 complete [ 4801.507137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.517657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4801.528642] Lustre: Skipped 1 previous similar message [ 4801.551420] Lustre: Skipped 1 previous similar message [ 4806.629233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4806.637350] Lustre: Skipped 1 previous similar message [ 4811.744086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.758127] Lustre: Skipped 1 previous similar message [ 4813.281084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4813.442919] Lustre: server umount lustre-MDT0000 complete [ 4816.864982] LustreError: 186768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.890966] LustreError: 186768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4817.440347] LustreError: 184253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102569 with bad export cookie 12806150398671276118 [ 4817.443377] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4817.451266] LustreError: 184253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4817.682907] Lustre: server umount lustre-MDT0001 complete [ 4831.086382] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4834.306268] Key type lgssc unregistered [ 4834.859598] LNet: 187831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4834.865223] LNetError: 187831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4834.884872] LNet: Removed LNI 192.168.203.147@tcp [ 4835.940573] Key type .llcrypt unregistered [ 4835.943125] Key type ._llcrypt unregistered [ 4864.051732] Key type ._llcrypt registered [ 4864.054084] Key type .llcrypt registered [ 4864.130072] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4864.682261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4864.722603] alg: No test for adler32 (adler32-zlib) [ 4865.754490] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 4865.982205] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4867.608260] Key type lgssc registered [ 4868.475204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4875.491848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4875.523044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.963565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.132900] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4886.561655] LustreError: 189267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.605672] LustreError: 189267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4889.816102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.135633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4895.628289] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4904.646867] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4912.028779] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4918.592398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.948549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.026903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4924.882403] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4932.112858] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.750828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.752225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.766793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.790899] Lustre: Skipped 1 previous similar message [ 4940.396378] Lustre: lustre-OST0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 4940.403754] Lustre: Skipped 1 previous similar message [ 4944.867920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.879976] Lustre: Skipped 1 previous similar message [ 4945.670774] Lustre: server umount lustre-OST0000 complete [ 4955.588240] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90b25032e800.ost_server_uuid 50 [ 4958.159310] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4974.561815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.565685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.586488] Lustre: Skipped 1 previous similar message [ 4974.604363] Lustre: Skipped 2 previous similar messages [ 4979.684269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.701584] Lustre: Skipped 1 previous similar message [ 4979.789621] Lustre: server umount lustre-MDT0000 complete [ 4983.998026] LustreError: 189248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102735 with bad export cookie 6345328983743200473 [ 4984.006040] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.014173] LustreError: 189248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4984.243882] Lustre: server umount lustre-MDT0001 complete [ 4994.875986] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:45:45 (1790102745) [ 5002.883577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.545662] LustreError: 193188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.665083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.879843] LustreError: 193189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.329448] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5013.989480] LustreError: 193188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.669768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.065272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5023.686808] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5032.424525] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5038.787422] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5045.834256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5046.286240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.362430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5052.599850] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5060.969535] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5067.981938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5068.216671] Lustre: lustre-OST0001: new disk, initializing [ 5068.220184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5068.229821] Lustre: 196030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5068.332920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5075.402556] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5078.105305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5078.118610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5078.210436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5085.029560] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5088.744827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.747931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5088.756282] Lustre: Skipped 1 previous similar message [ 5094.044100] Lustre: server umount lustre-OST0000 complete [ 5095.986772] LustreError: 195013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.126981] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5100.348299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5104.099496] LustreError: 195013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.116221] LustreError: 195013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5104.873800] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5105.115652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5124.584922] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.586146] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5124.587529] LustreError: 195010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.587550] LustreError: 195010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5124.615670] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5124.651913] Lustre: Skipped 5 previous similar messages [ 5126.702946] Lustre: server umount lustre-OST0001 complete [ 5131.745882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5131.753517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.769442] Lustre: Skipped 1 previous similar message [ 5137.635424] Lustre: server umount lustre-MDT0000 complete [ 5141.626604] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790102893 with bad export cookie 6345328983743201537 [ 5141.628593] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5141.645273] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5142.124234] Lustre: server umount lustre-MDT0001 complete [ 5169.243405] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:48:40 (1790102920) [ 5177.704828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.305220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5178.334554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5178.620687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.883079] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5193.916641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.114933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5194.155918] Lustre: 199470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5201.158594] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5212.565471] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5220.999923] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5228.499541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.645646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5234.304468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5236.090545] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5244.361913] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5251.516650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.613077] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5251.938380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.941538] Lustre: Skipped 2 previous similar messages [ 5258.581719] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5261.444742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5265.420319] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5269.984823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5269.994922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.006019] Lustre: Skipped 1 previous similar message [ 5270.009609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5270.014374] Lustre: Skipped 3 previous similar messages [ 5275.472689] Lustre: server umount lustre-OST0000 complete [ 5276.717945] LustreError: 201323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5276.745563] LustreError: 201323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5297.640700] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5297.644749] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.648059] LustreError: Skipped 1 previous similar message [ 5297.681546] Lustre: Skipped 2 previous similar messages [ 5299.545911] Lustre: server umount lustre-OST0001 complete [ 5307.363938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5311.434912] Lustre: server umount lustre-MDT0000 complete [ 5315.968205] LustreError: 199461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103067 with bad export cookie 6345328983743202951 [ 5315.981871] LustreError: 199461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5315.982309] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.654288] Lustre: server umount lustre-MDT0001 complete [ 5344.670770] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:51:35 (1790103095) [ 5359.375911] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 5369.198742] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 5411.396122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.722596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.760241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.878626] Lustre: lustre-MDT0000: new disk, initializing [ 5411.952388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.995146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5416.615891] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5427.158931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.289280] Lustre: 207566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5427.330676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5427.333924] Lustre: Skipped 1 previous similar message [ 5427.424851] Lustre: lustre-MDT0001: new disk, initializing [ 5427.568778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5427.586572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5432.293331] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5442.322901] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5449.339726] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5455.953487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.197564] Lustre: lustre-OST0000: new disk, initializing [ 5456.203362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5456.209478] Lustre: 209537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5462.126323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5462.140504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5462.247611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5463.321692] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5473.468544] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5478.847222] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5479.152927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5484.128398] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5484.406309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5488.106176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.106493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.121555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.121563] Lustre: Skipped 8 previous similar messages [ 5488.124612] LustreError: Skipped 1 previous similar message [ 5488.176155] Lustre: Skipped 3 previous similar messages [ 5492.689715] Lustre: server umount lustre-OST0000 complete [ 5502.832367] Lustre: server umount lustre-MDT0000 complete [ 5503.464427] LustreError: 208696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.490821] LustreError: 208696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 5507.276842] LustreError: 207561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103259 with bad export cookie 6345328983743204736 [ 5507.277728] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.290191] LustreError: 207561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.627669] Lustre: server umount lustre-MDT0001 complete [ 5516.686343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.648425] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5532.220293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.742282] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5546.955028] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5554.286801] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5563.304699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.654744] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5579.919571] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5585.323106] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5588.657529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5592.387743] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5592.572898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5598.690300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5598.693681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.695155] LustreError: Skipped 1 previous similar message [ 5598.714673] Lustre: Skipped 3 previous similar messages [ 5601.230796] Lustre: server umount lustre-OST0000 complete [ 5609.170324] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5614.477720] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5644.232247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.136371] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5658.593961] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5661.162471] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5664.012475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.031971] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.032038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5664.057641] Lustre: Skipped 1 previous similar message [ 5667.138209] Lustre: server umount lustre-OST0000 complete [ 5674.506563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5674.523698] Lustre: Skipped 6 previous similar messages [ 5682.786208] Lustre: server umount lustre-MDT0000 complete [ 5684.737472] LustreError: 211753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.788078] LustreError: 211753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5685.727472] Lustre: 188570:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103421/real 1790103421] req@00000000cba86eca x1877058641982720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790103437 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5685.802343] Lustre: 188570:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5689.161558] LustreError: 211739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103441 with bad export cookie 6345328983743205618 [ 5689.164151] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.178794] LustreError: 211739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5689.737882] Lustre: server umount lustre-MDT0001 complete [ 5705.267316] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:57:35 (1790103455) [ 5716.153883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.948746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.956139] Lustre: Skipped 6 previous similar messages [ 5723.352922] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5734.033525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.735470] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5749.352503] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5755.897379] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5763.835911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5770.894230] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5780.193114] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.394380] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5788.985372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5794.169272] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5794.333725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5800.632706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.865974] Lustre: lustre-OST0001: new disk, initializing [ 5800.869055] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5800.873384] Lustre: 221141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5808.215979] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5808.763137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5808.770735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5808.886147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5818.100033] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5822.898684] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5823.151907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5828.044451] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5828.246265] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5834.720683] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5834.727834] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.730672] LustreError: Skipped 1 previous similar message [ 5834.753555] Lustre: Skipped 4 previous similar messages [ 5836.410329] Lustre: server umount lustre-OST0001 complete [ 5843.551627] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5843.740715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5848.781227] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5849.042204] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5877.565856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.855667] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5882.868313] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5882.879594] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5883.750942] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5890.837386] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5892.455164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5900.483508] Lustre: server umount lustre-OST0001 complete [ 5930.975164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5931.157107] Lustre: server umount lustre-OST0000 complete [ 5939.170634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5939.176402] Lustre: Skipped 13 previous similar messages [ 5940.492485] Lustre: server umount lustre-MDT0000 complete [ 5943.706846] LustreError: 221140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103695 with bad export cookie 6345328983743206584 [ 5943.717813] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.726082] LustreError: 221140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5958.111203] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5958.345315] Lustre: server umount lustre-MDT0001 complete [ 5980.034826] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:02:11 (1790103731) [ 5985.452953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.768396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5985.805097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5989.827456] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5996.621508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.684767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5996.700454] Lustre: 226398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6000.164687] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6006.142809] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6010.737180] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6016.596850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.806768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6023.622925] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6031.048594] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6038.687787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.794917] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6044.387792] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6050.599905] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6051.866485] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6055.845728] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6056.089562] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6059.594973] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6059.768733] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6064.515236] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90b247c79000.ost_server_uuid 50 [ 6065.721599] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90b247c79000.ost_server_uuid in IDLE state after 0 sec [ 6066.440850] Lustre: Permanently deactivating lustre-OST0001 [ 6066.452924] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6069.304986] Lustre: Permanently reactivating lustre-OST0001 [ 6076.385052] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.390274] LustreError: Skipped 4 previous similar messages [ 6078.669634] Lustre: server umount lustre-OST0001 complete [ 6082.030664] LustreError: 228244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.041523] LustreError: 228244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6095.839130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6095.999955] Lustre: server umount lustre-OST0000 complete [ 6099.424097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.430077] Lustre: Skipped 10 previous similar messages [ 6104.894467] Lustre: server umount lustre-MDT0000 complete [ 6107.803132] LustreError: 226391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103859 with bad export cookie 6345328983743207872 [ 6107.804500] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.817081] LustreError: 226391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6108.056519] Lustre: server umount lustre-MDT0001 complete [ 6125.103870] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:04:36 (1790103876) [ 6130.475121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.684687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6130.712708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6130.723839] Lustre: Skipped 5 previous similar messages [ 6134.095419] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6140.906757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.983546] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6141.009581] Lustre: 233087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6144.803501] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6151.296434] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6155.689315] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6161.232383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6161.453072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6161.481189] Lustre: Permanently deactivating lustre-OST0000 [ 6161.502453] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6167.095525] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6169.092609] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6173.083460] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6178.380415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.459066] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6182.810954] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6188.699679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6188.968347] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6190.323507] Lustre: Permanently reactivating lustre-OST0000 [ 6190.342245] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6190.347151] Lustre: Skipped 2 previous similar messages [ 6194.721917] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6194.727566] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.741259] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6194.748375] Lustre: Skipped 1 previous similar message [ 6194.765979] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6194.779987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6205.091375] Lustre: server umount lustre-OST0001 complete [ 6214.309829] Lustre: server umount lustre-OST0000 complete [ 6224.421111] Lustre: server umount lustre-MDT0000 complete [ 6230.755945] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103982 with bad export cookie 6345328983743209517 [ 6230.761948] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6230.767089] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.948391] Lustre: server umount lustre-MDT0001 complete [ 6250.009698] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6254.278089] Key type lgssc unregistered [ 6255.357766] LNet: 237987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.363360] LNetError: 237987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6255.377077] LNet: Removed LNI 192.168.203.147@tcp [ 6256.461322] Key type .llcrypt unregistered [ 6256.466181] Key type ._llcrypt unregistered [ 6286.656730] Key type ._llcrypt registered [ 6286.660377] Key type .llcrypt registered [ 6286.791054] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6287.788694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6287.807477] alg: No test for adler32 (adler32-zlib) [ 6288.863366] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 6289.118615] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 6290.759939] Key type lgssc registered [ 6292.059681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6301.667953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6301.695741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.082559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.415405] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6313.459096] LustreError: 239469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.506290] LustreError: 239469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6316.284036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.393743] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6316.426994] Lustre: 239457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6316.436060] Lustre: Permanently deactivating lustre-MDT0001 [ 6316.450382] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6316.747619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6321.047970] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6321.675993] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6321.680346] LustreError: 239499:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6321.690266] LustreError: 239499:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6321.695864] LustreError: 239499:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 6327.975501] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.778587] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6340.595389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.976510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.015465] LustreError: 241311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.040254] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6347.119183] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6347.257885] LustreError: 241311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.356637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6353.379199] LustreError: 241311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6353.407454] LustreError: 241311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6354.150197] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6358.497944] LustreError: 241384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.512559] LustreError: 241384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6359.679043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.947536] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6365.176214] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6365.177755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6365.703076] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6372.333465] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6376.374849] Lustre: Permanently reactivating lustre-MDT0001 [ 6376.393474] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6376.397827] Lustre: Skipped 2 previous similar messages [ 6379.426546] Lustre: lustre-MDT0001: Client 2483e3ca-f95b-4e9b-958c-9baac40d8292 (at 192.168.203.47@tcp) reconnecting [ 6386.639934] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6388.989477] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6395.105696] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6395.294533] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6400.002951] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6400.405778] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6403.173210] Lustre: Permanently deactivating lustre-MDT0001 [ 6403.198192] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6403.204774] Lustre: Skipped 2 previous similar messages [ 6412.478778] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6423.671436] Lustre: server umount lustre-MDT0000 complete [ 6424.546497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.552028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.586044] LustreError: 239469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.607188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.630782] Lustre: Skipped 1 previous similar message [ 6427.491217] LustreError: 239450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104179 with bad export cookie 14594296327904555156 [ 6427.502418] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.508671] LustreError: 239450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6427.803218] Lustre: server umount lustre-MDT0001 complete [ 6431.397021] Lustre: server umount lustre-OST0000 complete [ 6443.678283] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:09:54 (1790104194) [ 6454.051862] Lustre: server umount lustre-OST0001 complete [ 6460.584365] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 6472.176796] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6508.389034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.575207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6508.581268] Lustre: Skipped 2 previous similar messages [ 6508.610661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6508.660190] Lustre: lustre-MDT0000: new disk, initializing [ 6508.725200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.750620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6513.180518] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6524.477974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.621738] Lustre: 247774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6524.747487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6524.757793] Lustre: Skipped 1 previous similar message [ 6524.914919] Lustre: lustre-MDT0001: new disk, initializing [ 6525.024476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.066851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6525.085581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6530.398323] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6541.269725] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6548.412054] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.451776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.890396] Lustre: lustre-OST0000: new disk, initializing [ 6555.894989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6555.900668] Lustre: 249742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6556.008292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.879558] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6565.502929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6565.523246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6565.617122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6572.526082] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6575.283982] LustreError: 247781:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6584.575136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.814501] Lustre: lustre-OST0001: new disk, initializing [ 6584.820359] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6584.828489] Lustre: 250860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6585.351482] LustreError: 247781:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6585.390945] LustreError: 250514:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6590.583611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6590.601461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6590.751328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6591.608650] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6595.399262] LustreError: 250514:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6602.244903] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6604.541141] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6611.428305] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6611.436853] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.442123] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6611.456709] Lustre: Skipped 2 previous similar messages [ 6612.874598] Lustre: server umount lustre-OST0001 complete [ 6616.549537] LustreError: 249726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.577026] LustreError: 249726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6631.910994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.920071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6631.929839] Lustre: Skipped 1 previous similar message [ 6631.972241] Lustre: Skipped 2 previous similar messages [ 6637.034649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6637.044532] Lustre: Skipped 1 previous similar message [ 6642.159714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6642.182515] Lustre: Skipped 1 previous similar message [ 6644.191147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6644.515223] Lustre: server umount lustre-OST0000 complete [ 6652.385154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.390907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6652.398933] Lustre: Skipped 1 previous similar message [ 6652.410036] Lustre: Skipped 1 previous similar message [ 6655.309951] Lustre: server umount lustre-MDT0000 complete [ 6657.504472] LustreError: 247781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.542231] LustreError: 247781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6659.411328] LustreError: 247766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104411 with bad export cookie 14594296327904557970 [ 6659.420026] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6659.430492] LustreError: 247766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6665.801871] Lustre: server umount lustre-MDT0001 complete [ 6676.730758] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6680.274532] Key type lgssc unregistered [ 6680.680556] LNet: 252895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6680.685214] LNetError: 252895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6681.703532] LNet: Removed LNI 192.168.203.147@tcp [ 6682.857854] Key type .llcrypt unregistered [ 6682.860310] Key type ._llcrypt unregistered [ 6722.779441] Key type ._llcrypt registered [ 6722.783436] Key type .llcrypt registered [ 6722.916924] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6724.178818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6724.197196] alg: No test for adler32 (adler32-zlib) [ 6725.261642] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 6725.567907] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 6727.279183] Key type lgssc registered [ 6728.277390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6735.267932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6735.282305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.547876] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6736.559334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6736.673924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.013679] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6750.685087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.898596] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6750.986305] Lustre: 255118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6751.474084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6756.483122] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6764.981235] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6771.610684] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6779.075583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6779.322925] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6779.564608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.760832] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6788.256881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6795.021316] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6849.507187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6849.524591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6849.541012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6850.016257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6850.038457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.055939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6850.062690] Lustre: Skipped 1 previous similar message [ 6855.143535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6855.160949] Lustre: Skipped 1 previous similar message [ 6855.626133] Lustre: server umount lustre-OST0000 complete [ 6860.160120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6872.081348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.446907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.696415] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6880.635224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6880.666622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6885.801355] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6885.825783] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6885.935572] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6926.819634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.821624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6926.828197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6926.840419] Lustre: Skipped 1 previous similar message [ 6926.856817] LustreError: Skipped 1 previous similar message [ 6931.495758] Lustre: server umount lustre-OST0000 complete [ 6935.521850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6935.531789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.551849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6935.567070] Lustre: Skipped 1 previous similar message [ 6941.571497] Lustre: server umount lustre-MDT0000 complete [ 6945.197782] LustreError: 255109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104697 with bad export cookie 2038738897934394577 [ 6945.209220] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6945.211941] LustreError: 255109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6945.657805] Lustre: server umount lustre-MDT0001 complete [ 6957.860894] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6960.573886] Key type lgssc unregistered [ 6960.855232] LNet: 260547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6960.866436] LNetError: 260547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6960.883059] LNet: Removed LNI 192.168.203.147@tcp [ 6961.820498] Key type .llcrypt unregistered [ 6961.822969] Key type ._llcrypt unregistered [ 6987.546812] Key type ._llcrypt registered [ 6987.549511] Key type .llcrypt registered [ 6987.665959] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6988.671619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6988.692691] alg: No test for adler32 (adler32-zlib) [ 6989.742966] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 6989.961600] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 6991.655209] Key type lgssc registered [ 6992.814988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7000.532769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7000.569240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.957141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.599538] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7011.306854] LustreError: 261979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.330047] LustreError: 261979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7013.085868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7013.480679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.877432] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7026.771899] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7032.277687] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7038.288364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.599906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7044.871962] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7046.825382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7051.629269] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.747246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7072.747284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7072.754744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7072.768192] Lustre: Skipped 1 previous similar message [ 7077.374936] Lustre: server umount lustre-OST0000 complete [ 7082.979216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.986663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.998036] Lustre: Skipped 1 previous similar message [ 7083.012529] Lustre: Skipped 2 previous similar messages [ 7088.096422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.105665] Lustre: Skipped 1 previous similar message [ 7092.193851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7092.200734] Lustre: Skipped 1 previous similar message [ 7095.266202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7095.389798] Lustre: server umount lustre-MDT0000 complete [ 7097.316632] LustreError: 261979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.331946] LustreError: 261979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7098.630992] LustreError: 261961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104850 with bad export cookie 11391660669163751642 [ 7098.631966] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.647653] LustreError: 261961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7099.014820] Lustre: server umount lustre-MDT0001 complete [ 7108.703666] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7110.944333] Key type lgssc unregistered [ 7111.251280] LNet: 266280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7111.259131] LNetError: 266280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7111.288214] LNet: Removed LNI 192.168.203.147@tcp [ 7112.002075] Key type .llcrypt unregistered [ 7112.004555] Key type ._llcrypt unregistered [ 7127.898119] Key type ._llcrypt registered [ 7127.899385] Key type .llcrypt registered [ 7127.990849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7128.909297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.982273] alg: No test for adler32 (adler32-zlib) [ 7130.013208] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7130.201742] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7131.863204] Key type lgssc registered [ 7133.145918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7141.856512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7141.872820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.318340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.127607] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7153.487280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.634244] LustreError: 267519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.660927] LustreError: 267519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7153.818781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7157.693933] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7164.464249] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7170.663991] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7177.132559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.500805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.881736] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7188.851889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7191.852678] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.327813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7204.329258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.337076] LustreError: Skipped 1 previous similar message [ 7204.338633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7204.379951] Lustre: Skipped 1 previous similar message [ 7204.564515] Lustre: server umount lustre-OST0000 complete [ 7209.440765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7209.457252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.461874] Lustre: Skipped 1 previous similar message [ 7209.482330] Lustre: Skipped 2 previous similar messages [ 7214.470728] Lustre: server umount lustre-MDT0000 complete [ 7214.560192] LustreError: 267519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.576833] LustreError: 267519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7217.824353] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104969 with bad export cookie 15404798107092700898 [ 7217.827033] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.833539] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7218.129979] Lustre: server umount lustre-MDT0001 complete [ 7227.446609] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7230.471677] Key type lgssc unregistered [ 7230.787851] LNet: 271192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7230.802418] LNetError: 271192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7230.817222] LNet: Removed LNI 192.168.203.147@tcp [ 7231.584168] Key type .llcrypt unregistered [ 7231.587426] Key type ._llcrypt unregistered [ 7254.381584] Key type ._llcrypt registered [ 7254.384283] Key type .llcrypt registered [ 7254.492761] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7255.660260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.813829] alg: No test for adler32 (adler32-zlib) [ 7256.918730] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7257.270379] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7258.967241] Key type lgssc registered [ 7260.148899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7267.387382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7267.424109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.813821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.295273] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7278.054094] LustreError: 272625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.072841] LustreError: 272625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7280.952315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.316726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7285.445395] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7292.340787] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7297.180176] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7304.188028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7304.442154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.600034] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7313.642870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7315.878696] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.495655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.504946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.507148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.532226] Lustre: Skipped 1 previous similar message [ 7344.629378] Lustre: server umount lustre-OST0000 complete [ 7349.738131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.742882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7349.758510] Lustre: Skipped 1 previous similar message [ 7349.780169] Lustre: Skipped 2 previous similar messages [ 7354.850307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7354.862716] Lustre: Skipped 1 previous similar message [ 7355.095265] Lustre: server umount lustre-MDT0000 complete [ 7357.996067] LustreError: 272605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105109 with bad export cookie 11554180706134184448 [ 7357.996851] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.010945] LustreError: 272605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7358.232872] Lustre: server umount lustre-MDT0001 complete [ 7366.448367] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7368.464968] Key type lgssc unregistered [ 7368.807377] LNet: 276926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7368.813426] LNetError: 276926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7368.828570] LNet: Removed LNI 192.168.203.147@tcp [ 7369.515328] Key type .llcrypt unregistered [ 7369.517945] Key type ._llcrypt unregistered [ 7385.332748] Key type ._llcrypt registered [ 7385.334834] Key type .llcrypt registered [ 7385.452704] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7386.214407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7386.304051] alg: No test for adler32 (adler32-zlib) [ 7387.351504] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7387.721524] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7389.415306] Key type lgssc registered [ 7390.859574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7399.439539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7399.470210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.073264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.174331] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7410.661263] LustreError: 278169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.676563] LustreError: 278169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7411.205984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.744646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.429452] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7421.733374] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7426.787398] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7432.339054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7432.649842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.460138] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7445.673554] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.026581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7456.224207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7456.235900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.245886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7456.255074] Lustre: Skipped 1 previous similar message [ 7461.348781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7461.358233] Lustre: Skipped 1 previous similar message [ 7461.729666] Lustre: server umount lustre-OST0000 complete [ 7466.471158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7466.475924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7466.488047] Lustre: Skipped 2 previous similar messages [ 7466.507208] Lustre: Skipped 1 previous similar message [ 7470.851104] Lustre: server umount lustre-MDT0000 complete [ 7471.589218] LustreError: 278175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.608526] LustreError: 278175:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7474.609316] LustreError: 278135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105226 with bad export cookie 17245671976610395980 [ 7474.614692] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7474.639034] LustreError: 278135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7475.057915] Lustre: server umount lustre-MDT0001 complete [ 7485.262463] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7488.006443] Key type lgssc unregistered [ 7488.325614] LNet: 281873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.344921] LNetError: 281873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.381754] LNet: Removed LNI 192.168.203.147@tcp [ 7489.216541] Key type .llcrypt unregistered [ 7489.218676] Key type ._llcrypt unregistered [ 7502.119174] Key type ._llcrypt registered [ 7502.134764] Key type .llcrypt registered [ 7502.284202] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7519.512880] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7534.250747] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7535.270275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.380098] alg: No test for adler32 (adler32-zlib) [ 7536.550524] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7536.831624] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7538.479808] Key type lgssc registered [ 7539.500904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7571.140213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7571.163398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.448731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7572.490243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7572.588393] Lustre: lustre-MDT0000: new disk, initializing [ 7572.651215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.666476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7577.341918] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7587.237455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.440497] Lustre: 285438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7587.545877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7587.555753] Lustre: Skipped 1 previous similar message [ 7587.638307] Lustre: lustre-MDT0001: new disk, initializing [ 7587.745222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7587.787564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7587.798951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7592.492768] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7601.948838] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7607.455870] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7613.528681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.766306] Lustre: lustre-OST0000: new disk, initializing [ 7613.773499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7613.777781] Lustre: 287412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7613.830399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.032914] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7620.158068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7620.179206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7620.279513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7626.545827] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7630.270370] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7630.445486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7634.673993] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7634.856498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7641.061634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.072698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7641.079155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7641.094207] Lustre: Skipped 1 previous similar message [ 7642.697567] Lustre: server umount lustre-OST0000 complete [ 7646.177876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.193139] Lustre: Skipped 1 previous similar message [ 7646.197314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.212150] Lustre: Skipped 1 previous similar message [ 7651.298609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.302864] Lustre: Skipped 1 previous similar message [ 7652.191249] Lustre: server umount lustre-MDT0000 complete [ 7655.089438] LustreError: 287411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105407 with bad export cookie 16128297531838345367 [ 7655.090019] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.103573] LustreError: 287411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7655.373788] Lustre: server umount lustre-MDT0001 complete [ 7662.408494] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:30:13 (1790105413) [ 7667.706744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.061389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.438486] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7673.316432] LustreError: 289824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.328717] LustreError: 289824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7677.408683] LustreError: 289823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.315489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.643148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.648432] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7690.061657] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7695.471758] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7702.975377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.226132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.126718] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7715.431893] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7725.025462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7725.035539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.043352] Lustre: Skipped 1 previous similar message [ 7725.048539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7728.104878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.117283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7731.124218] Lustre: server umount lustre-OST0000 complete [ 7735.264455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7735.274444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.291567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.297778] Lustre: Skipped 2 previous similar messages [ 7740.145342] Lustre: server umount lustre-MDT0000 complete [ 7743.461521] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.471289] LustreError: 289818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7743.593121] LustreError: 289803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105495 with bad export cookie 16128297531838346249 [ 7743.598789] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.604440] LustreError: 289803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.931919] Lustre: server umount lustre-MDT0001 complete [ 7756.316275] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7759.869745] Key type lgssc unregistered [ 7760.286528] LNet: 293359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7760.300361] LNetError: 293359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7760.327748] LNet: Removed LNI 192.168.203.147@tcp [ 7761.301842] Key type .llcrypt unregistered [ 7761.304334] Key type ._llcrypt unregistered [ 7785.350845] Key type ._llcrypt registered [ 7785.353329] Key type .llcrypt registered [ 7785.451543] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7802.027024] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7803.476167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7803.593547] alg: No test for adler32 (adler32-zlib) [ 7804.754876] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7805.183510] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7806.975247] Key type lgssc registered [ 7808.394593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7854.863904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7854.908347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.111785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7856.139249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7856.193197] Lustre: lustre-MDT0000: new disk, initializing [ 7856.238605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.253757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7860.146748] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7869.428290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.557546] Lustre: 297199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7869.603261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7869.606602] Lustre: Skipped 1 previous similar message [ 7869.664498] Lustre: lustre-MDT0001: new disk, initializing [ 7869.752975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7869.790897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7869.805924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7873.558031] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7881.760970] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7886.895144] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7893.094533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.379800] Lustre: lustre-OST2710: new disk, initializing [ 7893.389503] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7893.396947] Lustre: 299167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7893.484269] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7899.677847] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7902.297571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7902.315355] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7902.400904] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7908.967996] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7915.650371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7915.866267] Lustre: lustre-OST03e8: new disk, initializing [ 7915.870835] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7915.879293] Lustre: 300232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7915.989841] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7921.988611] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7922.772192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7922.779183] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7922.859674] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7929.151725] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7935.836642] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7936.027302] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.236032] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7941.391431] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7946.637726] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7946.915427] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7950.782692] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7950.978280] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7956.960615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7956.981215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.996368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.521616] Lustre: server umount lustre-MDT0000 complete [ 7959.035136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.053117] Lustre: Skipped 2 previous similar messages [ 7960.795988] LustreError: 297190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105712 with bad export cookie 8835886294728102615 [ 7960.799898] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.806038] LustreError: 297190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7961.038546] Lustre: server umount lustre-MDT0001 complete [ 7975.015153] Lustre: server umount lustre-OST2710 complete [ 7989.002843] Lustre: server umount lustre-OST03e8 complete [ 8005.495581] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8014.290110] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8048.255960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.462080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8048.479442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8048.575752] Lustre: lustre-MDT0000: new disk, initializing [ 8048.703693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.715230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8053.264221] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8064.283903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.436811] Lustre: 305479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8064.490535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8064.494755] Lustre: Skipped 1 previous similar message [ 8064.556541] Lustre: lustre-MDT0001: new disk, initializing [ 8064.627736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8064.660152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8064.677954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8068.589834] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8078.393564] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8083.807929] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8090.750667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.050194] Lustre: lustre-OST0000: new disk, initializing [ 8091.056122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8091.072930] Lustre: 307450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8091.212269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.132336] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8099.386741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8099.403856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8099.508648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8106.258494] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8110.765971] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8110.997297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8115.701945] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8115.917724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8120.294451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8120.295866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.298692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8120.312299] LustreError: Skipped 1 previous similar message [ 8120.354311] Lustre: Skipped 1 previous similar message [ 8125.411721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8125.418650] Lustre: Skipped 2 previous similar messages [ 8130.536680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8130.543020] Lustre: Skipped 1 previous similar message [ 8132.063689] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8132.165655] Lustre: server umount lustre-OST0000 complete [ 8135.649383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.668859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8135.674190] Lustre: Skipped 1 previous similar message [ 8135.692156] Lustre: Skipped 1 previous similar message [ 8142.489397] Lustre: server umount lustre-MDT0000 complete [ 8145.892502] LustreError: 305491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8145.911784] LustreError: 305491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8146.706864] LustreError: 305474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105898 with bad export cookie 8835886294728104295 [ 8146.709688] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.721909] LustreError: 305474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8147.166466] Lustre: server umount lustre-MDT0001 complete [ 8155.164740] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:38:26 (1790105906) [ 8156.653988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8158.614981] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:38:29 (1790105909) [ 8162.149841] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8189.585363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.898926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8189.927522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8190.181784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8194.965586] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8204.932211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8205.049571] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8205.079198] Lustre: 311305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8209.682273] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8231.547956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.789920] Lustre: Denying initial registration attempt from nid 192.168.203.147@tcp, specified as failover [ 8231.803386] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8231.816772] LustreError: 312762:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8231.822905] LustreError: 312762:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8231.830347] LustreError: 312762:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8231.917538] Lustre: server umount lustre-OST0000 complete [ 8231.921379] LustreError: 312762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8241.135853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8241.148792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.172392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8241.174622] Lustre: Skipped 2 previous similar messages [ 8244.784971] Lustre: server umount lustre-MDT0000 complete [ 8246.753277] LustreError: 311314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.779237] LustreError: 311314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8248.876521] LustreError: 311300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106000 with bad export cookie 8835886294728105170 [ 8248.881288] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.883402] LustreError: 311300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8249.145574] Lustre: server umount lustre-MDT0001 complete [ 8259.794530] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8262.391595] Key type lgssc unregistered [ 8262.711756] LNet: 313881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.740098] LNetError: 313881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8262.768496] LNet: Removed LNI 192.168.203.147@tcp [ 8263.692273] Key type .llcrypt unregistered [ 8263.694500] Key type ._llcrypt unregistered [ 8274.966408] Key type ._llcrypt registered [ 8274.969562] Key type .llcrypt registered [ 8275.102550] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8276.083671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8276.121380] alg: No test for adler32 (adler32-zlib) [ 8277.188060] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8277.433390] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8279.135195] Key type lgssc registered [ 8280.371418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8329.975966] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8338.498385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8338.538828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.938453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8339.973486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8340.197724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.271713] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8354.029372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8354.246127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8354.282272] Lustre: 316662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8354.656160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8359.450165] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8367.405208] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8373.204691] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8393.966754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.267551] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8394.607481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.955623] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8409.163558] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8414.180220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8414.196582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.214315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.694708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8414.706101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.723304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8417.579620] Lustre: server umount lustre-OST0000 complete [ 8421.354451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8421.359385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.390626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.929997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.931469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8427.476140] Lustre: server umount lustre-MDT0000 complete [ 8430.876388] LustreError: 316652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106182 with bad export cookie 6474086332373749721 [ 8430.886774] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8430.888759] LustreError: 316652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8431.208280] Lustre: server umount lustre-MDT0001 complete [ 8440.878689] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8443.403825] Key type lgssc unregistered [ 8443.652894] LNet: 320235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8443.660552] LNetError: 320235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8444.714757] LNet: Removed LNI 192.168.203.147@tcp [ 8445.579846] Key type .llcrypt unregistered [ 8445.581609] Key type ._llcrypt unregistered [ 8470.020252] Key type ._llcrypt registered [ 8470.022553] Key type .llcrypt registered [ 8470.116862] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8471.236799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8471.306634] alg: No test for adler32 (adler32-zlib) [ 8472.382640] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8472.794294] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8474.583441] Key type lgssc registered [ 8476.222100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8487.516068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8487.551148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.032139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.576745] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8498.145314] LustreError: 321667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.171063] LustreError: 321667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8503.268915] LustreError: 321666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.943020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.229044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8509.680688] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8517.232458] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8523.065599] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8531.755255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.160805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.071958] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8551.146872] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8561.120525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.124526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.133071] Lustre: Skipped 1 previous similar message [ 8561.148318] Lustre: Skipped 2 previous similar messages [ 8563.808416] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 8566.242184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8566.247448] Lustre: Skipped 2 previous similar messages [ 8567.446442] Lustre: server umount lustre-MDT0000 complete [ 8568.884675] LustreError: 321662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8570.704204] LustreError: 321649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106322 with bad export cookie 15540012427861548945 [ 8570.710063] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8570.713090] LustreError: 321649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8571.362345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.363527] LustreError: 324112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8571.382233] Lustre: Skipped 1 previous similar message [ 8571.418216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8575.393910] LustreError: 321663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8575.410871] LustreError: 321663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8579.118947] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 8579.130942] Lustre: Skipped 2 previous similar messages [ 8584.242309] LustreError: 321663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8584.258218] LustreError: 321663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8585.183165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8585.499059] Lustre: server umount lustre-MDT0001 complete [ 8591.652415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8599.800332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8615.458492] LustreError: 320966:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000859b7ecc x1877062423971328/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8615.759498] LustreError: 323501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.775436] LustreError: 323501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8615.828144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8615.913706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8619.685416] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8626.813049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.199646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8631.661564] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8635.389171] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8638.312554] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8641.717119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8646.246697] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8647.633093] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8652.772121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.778456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.792384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8652.797517] Lustre: Skipped 2 previous similar messages [ 8655.841772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.283659] Lustre: server umount lustre-OST0000 complete [ 8663.008681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.015579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.527131] Lustre: server umount lustre-MDT0000 complete [ 8670.880806] LustreError: 322403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106422 with bad export cookie 15540012427861556092 [ 8670.884356] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.896735] LustreError: 322403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8671.141782] Lustre: server umount lustre-MDT0001 complete [ 8679.563624] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8681.720752] Key type lgssc unregistered [ 8681.988685] LNet: 327939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8681.997125] LNetError: 327939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8682.017285] LNet: Removed LNI 192.168.203.147@tcp [ 8682.738694] Key type .llcrypt unregistered [ 8682.743385] Key type ._llcrypt unregistered [ 8702.187460] Key type ._llcrypt registered [ 8702.189454] Key type .llcrypt registered [ 8702.293118] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8703.030490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8703.112060] alg: No test for adler32 (adler32-zlib) [ 8704.109127] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8704.324734] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8705.975541] Key type lgssc registered [ 8707.112198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8713.094371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8713.124936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.454806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.044904] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8724.715166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.999686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8729.159445] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8735.226684] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8739.198844] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8744.423380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8744.429159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8744.449720] Lustre: Skipped 1 previous similar message [ 8744.465239] Lustre: Skipped 1 previous similar message [ 8747.479437] Lustre: server umount lustre-MDT0000 complete [ 8749.537795] LustreError: 330078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.557793] LustreError: 330078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8750.432988] LustreError: 329332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106502 with bad export cookie 8913336673877067791 [ 8750.439539] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8750.450635] LustreError: 329332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8750.702216] Lustre: server umount lustre-MDT0001 complete [ 8761.756992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.977454] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8761.995420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8762.166049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.730347] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8771.779666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.835599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8771.861491] Lustre: 331935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8771.919254] LustreError: 331947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8772.071782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8775.722961] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8781.576283] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8786.363285] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8794.598749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8794.734635] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8794.744241] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8794.751202] LustreError: 333819:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 8794.755080] LustreError: 333819:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8794.760569] LustreError: 333819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8794.808828] Lustre: server umount lustre-OST0000 complete [ 8794.813449] LustreError: 333819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8802.799121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.922264] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8803.103858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.571812] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8812.634687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8814.503894] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8820.155043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8820.207139] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8820.262867] Lustre: lustre-OST0001: new disk, initializing [ 8820.268291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8820.275530] Lustre: 335362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8820.350725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.038438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8821.060032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8821.143294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8825.146274] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8831.277576] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8836.579067] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8836.580846] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.587478] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8836.590167] LustreError: Skipped 1 previous similar message [ 8836.614121] Lustre: Skipped 1 previous similar message [ 8839.620698] Lustre: server umount lustre-OST0001 complete [ 8841.708625] LustreError: 334286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.725144] LustreError: 334286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8843.234042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8843.243588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.256912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8843.262533] Lustre: Skipped 1 previous similar message [ 8846.820021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8846.824625] LustreError: 334286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.826012] Lustre: Skipped 2 previous similar messages [ 8846.847814] LustreError: 334286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8848.605300] Lustre: server umount lustre-OST0000 complete [ 8851.936890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.942259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.947492] Lustre: Skipped 2 previous similar messages [ 8862.177938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8862.189611] Lustre: Skipped 4 previous similar messages [ 8866.271134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8866.410145] Lustre: server umount lustre-MDT0000 complete [ 8867.298171] LustreError: 331947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8867.314446] LustreError: 331947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8869.387790] LustreError: 334301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106621 with bad export cookie 8913336673877068344 [ 8869.391700] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8869.401101] LustreError: 334301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8869.732305] Lustre: server umount lustre-MDT0001 complete [ 8878.046276] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8879.975860] Key type lgssc unregistered [ 8880.189539] LNet: 337316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.197357] LNetError: 337316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8880.213151] LNet: Removed LNI 192.168.203.147@tcp [ 8880.832540] Key type .llcrypt unregistered [ 8880.835976] Key type ._llcrypt unregistered [ 8937.342288] Key type ._llcrypt registered [ 8937.345691] Key type .llcrypt registered [ 8937.426502] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8949.797198] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8950.981059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8951.023341] alg: No test for adler32 (adler32-zlib) [ 8952.092859] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8952.334374] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8954.047228] Key type lgssc registered [ 8955.196683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8988.216759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8988.243272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.460521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8989.498478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8989.602370] Lustre: lustre-MDT0000: new disk, initializing [ 8989.693554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8989.715690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8993.765249] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9002.867445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.952206] Lustre: 342242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9003.005499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9003.008318] Lustre: Skipped 1 previous similar message [ 9003.081284] Lustre: lustre-MDT0001: new disk, initializing [ 9003.162103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9003.191393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9003.201744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9006.798254] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9013.779360] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9018.348592] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9023.523288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9023.724812] Lustre: lustre-OST0000: new disk, initializing [ 9023.729792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9023.737798] Lustre: 344211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9023.816034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.786588] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9030.718687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9030.739579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9030.818617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9036.927744] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9040.958880] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9041.116805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9045.004841] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9045.169824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9051.621472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.621969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.636388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.646679] LustreError: Skipped 1 previous similar message [ 9056.740193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9056.744523] Lustre: Skipped 2 previous similar messages [ 9061.343113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9061.457243] Lustre: server umount lustre-OST0000 complete [ 9064.933241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9064.940224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.950285] Lustre: Skipped 1 previous similar message [ 9064.954555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9066.977525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9066.979250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9067.000250] Lustre: Skipped 1 previous similar message [ 9072.096365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.105772] Lustre: Skipped 1 previous similar message [ 9078.239137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9078.389093] Lustre: server umount lustre-MDT0000 complete [ 9080.885174] LustreError: 342236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106832 with bad export cookie 12684318138990406884 [ 9080.891754] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.895371] LustreError: 342236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9081.124838] Lustre: server umount lustre-MDT0001 complete [ 9087.772949] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:53:59 (1790106839) [ 9093.527768] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:54:05 (1790106845) [ 9100.128285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.470778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9103.296953] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9105.892027] LustreError: 346968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.906279] LustreError: 346968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9109.093435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9109.298468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.461933] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9117.925102] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9121.752872] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9126.445207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.693404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.365682] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9134.208675] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9155.000509] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9156.449144] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9157.602137] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9158.801172] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9160.144167] Lustre: Failing over lustre-MDT0000 [ 9160.267243] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 9160.273331] Lustre: Skipped 2 previous similar messages [ 9160.403209] Lustre: server umount lustre-MDT0000 complete [ 9160.677443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.679273] LustreError: 346967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9160.688628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9160.703412] Lustre: Skipped 2 previous similar messages [ 9160.724970] LustreError: 346967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9165.357369] LustreError: 346963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.480793] LustreError: 349273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.498117] LustreError: 349273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9175.607742] LustreError: 346964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.631634] LustreError: 346964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9177.198764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.278895] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9177.508985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.552866] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9180.722465] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9181.301320] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9182.694720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9182.707944] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9182.746045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9189.214634] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9190.378504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9193.331698] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9194.921817] Lustre: Failing over lustre-MDT0000 [ 9195.080675] Lustre: server umount lustre-MDT0000 complete [ 9196.100498] LustreError: 349273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.113693] LustreError: 349273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9198.049215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9198.053156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.065190] Lustre: Skipped 2 previous similar messages [ 9212.182838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.293544] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9212.535454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.587187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9215.909937] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9217.584122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9218.032082] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9218.041691] Lustre: Skipped 2 previous similar messages [ 9218.066939] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9218.109161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9223.449654] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9224.726627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9233.377691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.379412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9233.382378] Lustre: Skipped 1 previous similar message [ 9233.386789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9235.469107] Lustre: server umount lustre-MDT0000 complete [ 9238.369747] LustreError: 346947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106990 with bad export cookie 12684318138990409593 [ 9238.380078] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.504623] LustreError: 346964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.515134] LustreError: 346964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 9238.632563] Lustre: server umount lustre-MDT0001 complete [ 9247.722930] Lustre: server umount lustre-OST0000 complete [ 9260.342208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.432483] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9272.263489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.278456] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9282.628508] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9287.037603] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9292.935840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9293.267526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.272737] Lustre: Skipped 2 previous similar messages [ 9298.544965] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9304.246840] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.592143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9315.813193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.814764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.826155] Lustre: Skipped 3 previous similar messages [ 9315.834846] Lustre: Skipped 4 previous similar messages [ 9319.188933] Lustre: server umount lustre-OST0000 complete [ 9324.001642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9329.715134] Lustre: server umount lustre-MDT0000 complete [ 9331.169449] LustreError: 354053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.191677] LustreError: 354053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9333.401661] LustreError: 356106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107085 with bad export cookie 12684318138990410342 [ 9333.410270] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9333.421769] LustreError: 356106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9333.680349] Lustre: server umount lustre-MDT0001 complete [ 9344.906395] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9346.868594] Key type lgssc unregistered [ 9347.055610] LNet: 356856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9347.064090] LNetError: 356856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9347.074723] LNet: Removed LNI 192.168.203.147@tcp [ 9347.641170] Key type .llcrypt unregistered [ 9347.649816] Key type ._llcrypt unregistered [ 9370.250580] Key type ._llcrypt registered [ 9370.255990] Key type .llcrypt registered [ 9370.363138] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9371.262482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9371.313766] alg: No test for adler32 (adler32-zlib) [ 9372.379378] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 9372.613872] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9374.255253] Key type lgssc registered [ 9375.225517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9381.683043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9381.695886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.066157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.824130] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9393.631955] LustreError: 358367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.660315] LustreError: 358367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9393.939984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.246208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9397.582776] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9403.393614] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9407.589292] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9412.801822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.132715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.769821] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9423.035687] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9428.532168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9438.692198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.694558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9438.711713] Lustre: Skipped 2 previous similar messages [ 9438.720407] Lustre: Skipped 1 previous similar message [ 9442.752193] Lustre: server umount lustre-MDT0000 complete [ 9443.812169] LustreError: 358367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.831941] LustreError: 358367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9445.541255] LustreError: 358348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107197 with bad export cookie 14819615166384879411 [ 9445.545659] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.554207] LustreError: 358348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9445.778951] Lustre: server umount lustre-MDT0001 complete [ 9459.149940] Lustre: server umount lustre-OST0000 complete [ 9469.042388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.512701] LustreError: 362098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.569449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9473.024103] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9475.041186] LustreError: 362099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9479.143510] LustreError: 362098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9479.208217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.537968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9482.807398] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9487.967497] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.803277] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9496.324829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9496.546443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.010495] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9505.746971] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9514.993336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9530.337939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.341657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9530.358041] Lustre: Skipped 2 previous similar messages [ 9530.374028] Lustre: Skipped 2 previous similar messages [ 9532.678270] Lustre: server umount lustre-MDT0000 complete [ 9535.462478] LustreError: 363493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.484341] LustreError: 363493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9535.815405] LustreError: 362834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107287 with bad export cookie 14819615166384880524 [ 9535.815792] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9535.823625] LustreError: 362834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9536.106051] Lustre: server umount lustre-MDT0001 complete [ 9549.897519] Lustre: server umount lustre-OST0000 complete [ 9562.390559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.713450] LustreError: 366044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.766955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9566.339495] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9573.664077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9577.825502] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9584.971815] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9589.939428] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9595.360284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9595.640136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9595.646496] Lustre: Skipped 1 previous similar message [ 9600.500308] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9606.936671] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9607.923459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9618.411449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.415913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9618.421422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9622.810081] Lustre: server umount lustre-OST0000 complete [ 9628.644389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.647521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.659950] Lustre: Skipped 2 previous similar messages [ 9628.666201] Lustre: Skipped 1 previous similar message [ 9631.955383] Lustre: server umount lustre-MDT0000 complete [ 9633.761247] LustreError: 366779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.784316] LustreError: 366779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9634.988461] LustreError: 366024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107386 with bad export cookie 14819615166384881546 [ 9634.992040] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9634.998811] LustreError: 366024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9635.245416] Lustre: server umount lustre-MDT0001 complete [ 9644.438396] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9646.839246] Key type lgssc unregistered [ 9647.097325] LNet: 369574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9647.105103] LNetError: 369574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9647.121692] LNet: Removed LNI 192.168.203.147@tcp [ 9647.863765] Key type .llcrypt unregistered [ 9647.866286] Key type ._llcrypt unregistered [ 9681.013776] Key type ._llcrypt registered [ 9681.019562] Key type .llcrypt registered [ 9681.221912] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9682.646431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9682.659537] alg: No test for adler32 (adler32-zlib) [ 9685.646440] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 9686.387408] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9688.242141] Key type lgssc registered [ 9689.601017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9701.563275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9701.582697] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.593778] LustreError: 371041:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9701.610300] LustreError: 371041:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9701.618426] LustreError: 371041:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9701.630807] LustreError: 371041:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9701.640572] LustreError: 371041:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9713.768728] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9713.786447] LustreError: 371440:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9713.822646] LustreError: 371440:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9713.830899] LustreError: 371440:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9713.849666] LustreError: 371440:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9713.875883] LustreError: 371440:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9735.751461] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9738.799062] Key type lgssc unregistered [ 9739.120586] LNet: 372444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9739.129258] LNetError: 372444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9739.154035] LNet: Removed LNI 192.168.203.147@tcp [ 9740.005798] Key type .llcrypt unregistered [ 9740.009376] Key type ._llcrypt unregistered [ 9759.141709] Key type ._llcrypt registered [ 9759.143960] Key type .llcrypt registered [ 9759.265953] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9773.987305] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9775.246442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9775.261126] alg: No test for adler32 (adler32-zlib) [ 9776.326181] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 9776.636848] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9778.351211] Key type lgssc registered [ 9779.426843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9820.743870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9820.801210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.030676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9822.069444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9822.201550] Lustre: lustre-MDT0000: new disk, initializing [ 9822.370253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.387228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9827.553755] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9838.664197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.835913] Lustre: 375617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9838.903449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9838.906745] Lustre: Skipped 1 previous similar message [ 9838.997543] Lustre: lustre-MDT0001: new disk, initializing [ 9839.116457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9839.189463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9839.200709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9843.562408] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9853.416186] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9859.369415] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9865.916440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.149812] Lustre: lustre-OST0000: new disk, initializing [ 9866.154636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9866.163318] Lustre: 377587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9866.254618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.582583] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9873.991600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9874.004546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9874.177051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9882.583138] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9887.170799] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9887.436476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9892.215609] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9892.494437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9894.880507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.884760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.924616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.926264] Lustre: Skipped 1 previous similar message [ 9900.002091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.013072] Lustre: Skipped 2 previous similar messages [ 9900.780043] Lustre: server umount lustre-OST0000 complete [ 9905.121123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.153672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.166126] Lustre: Skipped 1 previous similar message [ 9910.255251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.264033] Lustre: Skipped 1 previous similar message [ 9910.581123] Lustre: server umount lustre-MDT0000 complete [ 9914.290611] LustreError: 375610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107666 with bad export cookie 112632593955675552 [ 9914.314219] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9914.636714] Lustre: server umount lustre-MDT0001 complete [ 9924.009766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9925.990848] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:07:56 (1790107676) [ 9933.134140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.783549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9938.675920] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9938.917840] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9938.940982] LustreError: 380049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9944.037377] LustreError: 380050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.624083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.953486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9951.126737] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9958.915257] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9965.636705] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9973.028267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.397400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.568923] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9988.014534] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.498242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.711097] Lustre: lustre-OST0001: new disk, initializing [ 9995.715406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9995.724910] Lustre: 382884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9995.842695] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10002.442070] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10005.078181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10005.090499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10005.166528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10012.299277] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10019.421733] Lustre: lustre-OST0001: Not available for connect from 192.168.203.47@tcp (stopping) [10020.842539] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10020.843180] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.868902] Lustre: Skipped 1 previous similar message [10022.401632] Lustre: server umount lustre-OST0001 complete [10024.493633] LustreError: 382695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.074981] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.109504] LustreError: 381873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10036.208205] LustreError: 382695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.225521] LustreError: 382695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10041.314774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.318834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10041.341359] Lustre: Skipped 1 previous similar message [10041.342206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10041.345715] LustreError: Skipped 1 previous similar message [10041.347294] Lustre: Skipped 2 previous similar messages [10045.313199] Lustre: server umount lustre-OST0000 complete [10049.504801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10049.513831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.531523] Lustre: Skipped 1 previous similar message [10055.548899] Lustre: server umount lustre-MDT0000 complete [10056.683437] LustreError: 381095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.708163] LustreError: 381095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10059.234311] LustreError: 380030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107811 with bad export cookie 112632593955676434 [10059.245356] LustreError: 380030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10059.251037] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10065.665732] Lustre: server umount lustre-MDT0001 complete [10076.457256] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10079.074660] Key type lgssc unregistered [10079.336610] LNet: 384872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10079.369845] LNetError: 384872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10080.423154] LNet: Removed LNI 192.168.203.147@tcp [10081.290082] Key type .llcrypt unregistered [10081.292344] Key type ._llcrypt unregistered [10135.660665] Key type ._llcrypt registered [10135.663255] Key type .llcrypt registered [10135.783407] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10136.880751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10136.917144] alg: No test for adler32 (adler32-zlib) [10137.953722] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10138.163243] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10139.815255] Key type lgssc registered [10140.982707] Lustre: Echo OBD driver; http://www.lustre.org/ [10147.740991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10147.778794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.097980] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10149.123413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10149.215109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10153.280585] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10161.647212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.777903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10161.808903] Lustre: 387691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10162.158695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.660211] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10174.940492] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.931745] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10186.215472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.221189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.235612] Lustre: Skipped 1 previous similar message [10186.245521] Lustre: Skipped 1 previous similar message [10190.185871] Lustre: server umount lustre-MDT0000 complete [10194.741503] LustreError: 387682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107946 with bad export cookie 13938959528178521382 [10194.762813] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.126218] Lustre: server umount lustre-MDT0001 complete [10204.556645] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:12:35 (1790107955) [10210.289188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.837631] LustreError: 390201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.862247] LustreError: 390201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10210.900814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.311888] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10215.930681] LustreError: 390202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.041706] LustreError: 390201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.945734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.403391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10229.120996] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10235.643536] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10240.974089] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10247.347407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10247.545703] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10247.842622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10254.092647] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10261.273646] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10268.368650] Lustre: Permanently deactivating lustre-OST0000 [10268.385467] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10277.899491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10280.627219] LustreError: 393284:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10280.643048] LustreError: 393284:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10280.655183] LustreError: 393284:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10284.047845] LustreError: 393333:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10284.061396] LustreError: 393333:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10292.017039] Lustre: server umount lustre-OST0000 complete [10294.315645] LustreError: 393582:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10294.325180] LustreError: 393582:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10296.299480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.304311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10296.317763] Lustre: Skipped 1 previous similar message [10296.318173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10296.349144] Lustre: Skipped 1 previous similar message [10301.410780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10301.418570] Lustre: Skipped 1 previous similar message [10306.531647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10306.538757] Lustre: Skipped 1 previous similar message [10306.589537] Lustre: server umount lustre-MDT0000 complete [10310.915409] LustreError: 390181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108062 with bad export cookie 13938959528178521963 [10310.920336] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.929101] LustreError: 390181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10320.553584] Lustre: server umount lustre-MDT0001 complete [10329.107739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.903856] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10336.506744] LustreError: 394741:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10342.479683] Lustre: 395080:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.147@tcp [10344.595947] Lustre: server umount MGS complete [10355.784323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.217419] LustreError: 395804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.251980] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10356.259988] Lustre: Skipped 1 previous similar message [10356.344518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10361.058605] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10361.843056] LustreError: 395805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.588378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10368.797473] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10372.606471] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10380.705839] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10387.870435] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10389.884484] Lustre: Permanently reactivating lustre-OST0000 [10389.919184] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10389.934102] Lustre: Skipped 2 previous similar messages [10415.598290] LustreError: 397536:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10423.413056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10423.758741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10423.763334] Lustre: Skipped 1 previous similar message [10431.076042] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10439.780546] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10446.304280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10446.309730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.315457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10446.324320] Lustre: Skipped 1 previous similar message [10446.338078] Lustre: Skipped 1 previous similar message [10451.927984] Lustre: server umount lustre-OST0000 complete [10456.037828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10456.044760] LustreError: Skipped 1 previous similar message [10456.048776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.068676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10456.081262] Lustre: Skipped 2 previous similar messages [10462.090230] Lustre: server umount lustre-MDT0000 complete [10465.963776] LustreError: 395784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108217 with bad export cookie 13938959528178523013 [10465.969904] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10465.978152] LustreError: 395784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10466.493292] Lustre: server umount lustre-MDT0001 complete [10477.639685] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10480.301533] Key type lgssc unregistered [10480.632647] LNet: 400123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10480.650774] LNetError: 400123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10480.686566] LNet: Removed LNI 192.168.203.147@tcp [10481.663799] Key type .llcrypt unregistered [10481.666238] Key type ._llcrypt unregistered [10499.082685] Key type ._llcrypt registered [10499.084352] Key type .llcrypt registered [10499.171076] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10514.448316] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10515.352114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10515.515839] alg: No test for adler32 (adler32-zlib) [10516.595424] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10516.870090] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10518.607235] Key type lgssc registered [10519.591711] Lustre: Echo OBD driver; http://www.lustre.org/ [10553.646392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10553.668647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.052032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10555.100169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10555.218695] Lustre: lustre-MDT0000: new disk, initializing [10555.280539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.307965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10560.713458] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10571.824039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10571.989144] Lustre: 403300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10572.109078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10572.118771] Lustre: Skipped 1 previous similar message [10572.197181] Lustre: lustre-MDT0001: new disk, initializing [10572.358249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10572.393266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10572.407262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10577.986961] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10589.429556] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10596.276923] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10603.699544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10604.093782] Lustre: lustre-OST0000: new disk, initializing [10604.098874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10604.108526] Lustre: 405268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10604.257373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10610.676721] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10611.795137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10611.814368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10611.925373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10620.261245] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10625.788239] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10626.122429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10631.713818] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10631.959593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10637.796667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10637.797137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10637.818478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10637.836648] LustreError: Skipped 1 previous similar message [10640.529848] Lustre: server umount lustre-OST0000 complete [10648.033890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10648.052173] Lustre: Skipped 1 previous similar message [10648.068076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10648.078267] Lustre: Skipped 2 previous similar messages [10653.152893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10653.158867] Lustre: Skipped 1 previous similar message [10657.249355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10657.260357] Lustre: Skipped 1 previous similar message [10659.295116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10659.452224] Lustre: server umount lustre-MDT0000 complete [10663.626078] LustreError: 403290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108415 with bad export cookie 7524594765732467969 [10663.626712] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10663.637021] LustreError: 403290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10664.000594] Lustre: server umount lustre-MDT0001 complete [10673.734953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10675.963743] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:20:26 (1790108426) [10683.939972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.462694] LustreError: 407731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.487152] LustreError: 407731:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10684.557542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.089982] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10690.026463] LustreError: 407732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.126750] LustreError: 407731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.175064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.545135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10702.714370] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10711.085982] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.517781] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10728.240335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10728.750712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10735.195224] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10742.527306] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10744.361278] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10747.008398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10755.554602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10755.569507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.580910] Lustre: Skipped 1 previous similar message [10755.592252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10757.603594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10757.605512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.913902] Lustre: server umount lustre-OST0000 complete [10764.256105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10764.267290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.292034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.299213] Lustre: Skipped 2 previous similar messages [10769.337540] Lustre: server umount lustre-MDT0000 complete [10772.970977] LustreError: 407727:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10773.001296] LustreError: 407727:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10773.195190] LustreError: 407713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108525 with bad export cookie 7524594765732468851 [10773.196876] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.209254] LustreError: 407713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10773.613378] Lustre: server umount lustre-MDT0001 complete [10784.734716] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10787.286259] Key type lgssc unregistered [10787.602639] LNet: 411455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10787.617426] LNetError: 411455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10787.637274] LNet: Removed LNI 192.168.203.147@tcp [10788.628184] Key type .llcrypt unregistered [10788.632227] Key type ._llcrypt unregistered [10810.646826] Key type ._llcrypt registered [10810.648749] Key type .llcrypt registered [10810.755915] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10811.012478] Key type .llcrypt unregistered [10811.016281] Key type ._llcrypt unregistered [10827.774988] Key type ._llcrypt registered [10827.776774] Key type .llcrypt registered [10827.903870] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10828.880407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10829.129407] alg: No test for adler32 (adler32-zlib) [10830.244578] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10830.539521] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10832.223547] Key type lgssc registered [10833.549082] Lustre: Echo OBD driver; http://www.lustre.org/ [10840.784854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10840.801266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.349476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.188615] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10852.839741] LustreError: 413684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.861536] LustreError: 413684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10855.555362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10856.048149] LustreError: 413683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.073636] LustreError: 413683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10856.077193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10861.541658] LustreError: 413684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.559924] LustreError: 413684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10862.345874] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10866.664071] LustreError: 413680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.688744] LustreError: 413680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10867.713820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10870.348468] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10872.802312] LustreError: 413683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.824556] LustreError: 413683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10877.692884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.922799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10877.930795] Lustre: Skipped 1 previous similar message [10878.041369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10882.369210] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10893.279918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10893.300797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10893.310759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10893.316202] Lustre: Skipped 1 previous similar message [10893.338308] Lustre: Skipped 1 previous similar message [10897.292255] Lustre: server umount lustre-OST0000 complete [10903.523891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.533392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10903.556301] Lustre: Skipped 1 previous similar message [10903.572847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10907.455945] Lustre: server umount lustre-MDT0000 complete [10908.640851] LustreError: 413679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10908.655764] LustreError: 413679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10911.053937] LustreError: 413662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108663 with bad export cookie 6710114085335975131 [10911.057440] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10911.071292] LustreError: 413662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10911.371670] Lustre: server umount lustre-MDT0001 complete [10921.782440] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [10924.605344] Key type lgssc unregistered [10924.889160] LNet: 416829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10924.895584] LNetError: 416829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10925.925824] LNet: Removed LNI 192.168.203.147@tcp [10927.049405] Key type .llcrypt unregistered [10927.052649] Key type ._llcrypt unregistered [10954.402463] Key type ._llcrypt registered [10954.404728] Key type .llcrypt registered [10954.546902] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10955.420793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10955.508904] alg: No test for adler32 (adler32-zlib) [10956.643274] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10956.946992] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [10958.639182] Key type lgssc registered [10959.677464] Lustre: Echo OBD driver; http://www.lustre.org/ [10966.426464] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10966.437328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.047374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10987.167304] LustreError: 418253:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [11013.920247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11019.558637] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11027.550994] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11033.748627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.081414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11034.141203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11037.954410] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11039.205624] LustreError: 418278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.232276] LustreError: 418278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11044.321605] LustreError: 418279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.594658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.849946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11048.312459] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11054.757604] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11059.616219] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11068.147802] Lustre: server umount lustre-OST0000 complete [11074.528495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.539192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11074.540307] Lustre: Skipped 1 previous similar message [11074.563164] Lustre: Skipped 1 previous similar message [11079.649473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11079.653525] Lustre: Skipped 1 previous similar message [11080.762581] Lustre: server umount lustre-MDT0000 complete [11083.771644] LustreError: 420029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108835 with bad export cookie 15771768007743580657 [11083.779192] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.784316] LustreError: 420029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11094.146910] Lustre: server umount lustre-MDT0001 complete [11101.843278] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11103.829147] Key type lgssc unregistered [11104.102722] LNet: 421823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11104.109718] LNetError: 421823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11104.124824] LNet: Removed LNI 192.168.203.147@tcp [11104.768983] Key type .llcrypt unregistered [11104.770906] Key type ._llcrypt unregistered [11123.134591] Key type ._llcrypt registered [11123.136956] Key type .llcrypt registered [11123.216890] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11124.286707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11124.300886] alg: No test for adler32 (adler32-zlib) [11125.354091] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11125.652326] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11127.399172] Key type lgssc registered [11128.400223] Lustre: Echo OBD driver; http://www.lustre.org/ [11134.740577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11134.762881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.117226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.345387] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11145.555543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.808253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11149.018189] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11154.065466] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11157.540328] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11161.938067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11162.179307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11166.347964] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11172.031766] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11181.614161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11186.658103] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.664586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11186.671903] Lustre: Skipped 2 previous similar messages [11186.679690] Lustre: Skipped 1 previous similar message [11186.687200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11189.316396] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [11189.324466] Lustre: Skipped 1 previous similar message [11191.777669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11191.786453] Lustre: Skipped 2 previous similar messages [11192.276976] Lustre: server umount lustre-MDT0000 complete [11193.751924] LustreError: 423968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.47@tcp arrived at 1790108945 with bad export cookie 16883790515393487332 [11193.764802] LustreError: 423968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11195.084763] LustreError: 423215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108947 with bad export cookie 16883790515393486667 [11195.097894] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11195.873903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11195.889416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.907339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11196.897551] LustreError: 423235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.911829] LustreError: 423235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11200.928434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11200.932212] LustreError: 423961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.935479] Lustre: Skipped 1 previous similar message [11200.950827] LustreError: 423961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11201.338871] Lustre: server umount lustre-OST0000 complete [11206.052252] LustreError: 423235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11206.062848] LustreError: 423235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11206.389963] Lustre: server umount lustre-MDT0001 complete [11213.276729] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11215.147768] Key type lgssc unregistered [11215.369459] LNet: 426938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11215.376408] LNetError: 426938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11216.424277] LNet: Removed LNI 192.168.203.147@tcp [11217.084829] Key type .llcrypt unregistered [11217.087326] Key type ._llcrypt unregistered [11233.243378] Key type ._llcrypt registered [11233.245746] Key type .llcrypt registered [11233.325952] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11234.021835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11234.058388] alg: No test for adler32 (adler32-zlib) [11234.999529] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11235.168561] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11236.791718] Key type lgssc registered [11237.631249] Lustre: Echo OBD driver; http://www.lustre.org/ [11242.642795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11242.663040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.965907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11247.142606] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11253.222449] LustreError: 428350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.245577] LustreError: 428350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11253.310710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.521213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11256.196675] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11260.662340] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11264.150107] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11268.336922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11268.548854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11272.308920] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11276.849555] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11288.047109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11294.688589] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11294.691729] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.697892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11297.863276] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [11298.277058] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.425250] Lustre: server umount lustre-MDT0001 complete [11303.396592] LustreError: 430644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.409517] LustreError: 430644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11308.515923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11308.515978] LustreError: 428350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.516077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.529078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11308.550438] Lustre: Skipped 2 previous similar messages [11310.187941] Lustre: server umount lustre-OST0000 complete [11312.608184] Lustre: server umount lustre-MDT0000 complete [11320.991537] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11322.681707] Key type lgssc unregistered [11322.890963] LNet: 432054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11322.900197] LNetError: 432054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11322.916570] LNet: Removed LNI 192.168.203.147@tcp [11323.467157] Key type .llcrypt unregistered [11323.468836] Key type ._llcrypt unregistered [11338.390858] Key type ._llcrypt registered [11338.392377] Key type .llcrypt registered [11338.452125] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [11348.246340] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11348.664961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11348.677593] alg: No test for adler32 (adler32-zlib) [11349.615705] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11349.747381] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11351.351215] Key type lgssc registered [11352.091676] Lustre: Echo OBD driver; http://www.lustre.org/ [11373.486846] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11381.512253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11381.521871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.651296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11382.668284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11382.710996] Lustre: lustre-MDT0000: new disk, initializing [11382.750046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11382.765817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11384.790581] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11393.461301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.606650] Lustre: lustre-OST0000: new disk, initializing [11393.609350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11393.611967] Lustre: Skipped 1 previous similar message [11393.615199] Lustre: 437075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11393.659196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11394.515201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11394.520472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11394.540244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11396.891809] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11402.354304] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11412.472537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11412.577110] Lustre: 436059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11412.612767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11412.668263] Lustre: lustre-MDT0001: new disk, initializing [11412.743640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11412.771055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11412.782499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11416.967909] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11429.575995] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11430.419627] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11436.407312] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11436.606374] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11444.194045] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11444.198553] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.204273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11444.208301] Lustre: Skipped 1 previous similar message [11447.070415] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [11449.320888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11449.361278] Lustre: Skipped 1 previous similar message [11449.467396] Lustre: server umount lustre-MDT0001 complete [11459.662650] LustreError: 436064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.688589] LustreError: 436064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11464.674487] LustreError: 436064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.684941] LustreError: 436064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11469.793171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.794574] LustreError: 436070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.808417] Lustre: Skipped 1 previous similar message [11469.815294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11473.873775] Lustre: server umount lustre-OST0000 complete [11474.914296] LustreError: 436069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11492.322181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11492.602481] Lustre: server umount lustre-MDT0000 complete [11510.434211] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11513.896640] Key type lgssc unregistered [11514.193265] LNet: 440823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11514.229711] LNetError: 440823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11514.250812] LNet: Removed LNI 192.168.203.147@tcp [11515.357372] Key type .llcrypt unregistered [11515.363764] Key type ._llcrypt unregistered [11555.820137] Key type ._llcrypt registered [11555.821769] Key type .llcrypt registered [11555.913307] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11556.254570] Key type .llcrypt unregistered [11556.258108] Key type ._llcrypt unregistered [11574.202882] Key type ._llcrypt registered [11574.206761] Key type .llcrypt registered [11574.333672] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11575.626040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.675377] alg: No test for adler32 (adler32-zlib) [11576.751672] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11577.032833] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11578.703723] Key type lgssc registered [11579.780993] Lustre: Echo OBD driver; http://www.lustre.org/ [11607.518902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11607.543896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.748520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11608.783749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11608.841724] Lustre: lustre-MDT0000: new disk, initializing [11608.966883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11608.986321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11613.021391] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11623.496252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11623.647288] Lustre: 444217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11623.715532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11623.720565] Lustre: Skipped 1 previous similar message [11623.838433] Lustre: lustre-MDT0001: new disk, initializing [11623.938039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11624.002742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11624.019800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11628.239440] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11637.601584] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11643.023519] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11650.606709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11650.783410] Lustre: lustre-OST0000: new disk, initializing [11650.785949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11650.791275] Lustre: 446186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11650.863708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11657.765255] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11657.834034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11657.849535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11657.956843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11668.115846] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11678.702282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.711232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.721904] Lustre: Skipped 1 previous similar message [11678.746795] Lustre: Skipped 2 previous similar messages [11681.860720] Lustre: server umount lustre-MDT0000 complete [11686.164057] LustreError: 444208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109438 with bad export cookie 15498133779063617769 [11686.168038] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.182370] LustreError: 444208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11686.729922] Lustre: server umount lustre-MDT0001 complete [11701.140800] Lustre: server umount lustre-OST0000 complete [11713.302884] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:37:43 (1790109463) [11721.895461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.500385] LustreError: 448215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.532647] LustreError: 448215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11722.578787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11726.838492] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11727.841853] LustreError: 448216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11732.960544] LustreError: 448215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.223308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11733.577531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11736.352865] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11740.738449] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11744.737623] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11762.085069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11762.223063] Lustre: Found index 0 for lustre-OST0000, updating log [11762.358167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11766.143475] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11767.470550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11770.697639] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11778.022740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.034026] Lustre: Skipped 1 previous similar message [11778.043771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11778.044567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11779.108070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11779.579179] Lustre: server umount lustre-OST0000 complete [11794.913339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11794.920768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.939134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11799.647974] Lustre: server umount lustre-MDT0000 complete [11802.070596] LustreError: 448198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109554 with bad export cookie 15498133779063619162 [11802.075692] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11802.081948] LustreError: 448198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11802.276160] Lustre: server umount lustre-MDT0001 complete [11807.624375] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:39:19 (1790109559) [11828.959252] LNet: 1 local NIs in recovery (showing 1): 192.168.203.147@tcp [12418.903056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12419.398038] LustreError: 454622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.478411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12424.129912] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12424.673531] LustreError: 454623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.797159] LustreError: 454622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.777758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.215254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12437.526986] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12445.398087] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12451.807882] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12472.131221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12472.413181] Lustre: Found index 0 for lustre-OST0000, updating log [12472.771141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12479.388677] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12481.513276] LNet: 1 local NIs in recovery (showing 1): 192.168.203.147@tcp [12487.447567] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12490.323481] Lustre: server umount lustre-OST0000 complete [12496.038960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.053840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.061037] Lustre: Skipped 2 previous similar messages [12496.074169] Lustre: Skipped 5 previous similar messages [12497.123037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12497.139042] Lustre: Skipped 1 previous similar message [12498.145846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.193714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12500.199341] Lustre: Skipped 4 previous similar messages [12501.399866] Lustre: server umount lustre-MDT0000 complete [12502.242442] LustreError: 454622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.262364] LustreError: 454622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12505.472725] LustreError: 454602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110257 with bad export cookie 15498133779063620226 [12505.477816] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12505.482809] LustreError: 454602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12505.964509] Lustre: server umount lustre-MDT0001 complete [12537.862895] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [12548.858734] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12585.611585] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:52:16 (1790110336) [12603.898624] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12606.529750] Key type lgssc unregistered [12606.756553] LNet: 461239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12606.766518] LNetError: 461239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12735.791785] LNet: Removed LNI 192.168.203.147@tcp [12736.755142] Key type .llcrypt unregistered [12736.756748] Key type ._llcrypt unregistered [12752.712716] Key type ._llcrypt registered [12752.714975] Key type .llcrypt registered [12752.808126] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12753.749236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12753.783085] alg: No test for adler32 (adler32-zlib) [12754.837155] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [12755.116096] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12756.815238] Key type lgssc registered [12758.134190] Lustre: Echo OBD driver; http://www.lustre.org/ [12762.139535] LNet: Added LNI 192.168.203.147@tcp100 [8/256/0/180] [12762.909028] LNet: Added LNI 192.168.203.147@tcp101 [8/256/0/180] [12763.729533] LNet: Added LNI 192.168.203.147@tcp102 [8/256/0/180] [12764.388223] LNet: Added LNI 192.168.203.147@tcp103 [8/256/0/180] [12765.212732] LNet: Added LNI 192.168.203.147@tcp104 [8/256/0/180] [12765.970454] LNet: Added LNI 192.168.203.147@tcp105 [8/256/0/180] [12766.709611] LNet: Added LNI 192.168.203.147@tcp106 [8/256/0/180] [12767.537364] LNet: Added LNI 192.168.203.147@tcp107 [8/256/0/180] [12768.215672] LNet: Added LNI 192.168.203.147@tcp108 [8/256/0/180] [12768.989493] LNet: Added LNI 192.168.203.147@tcp109 [8/256/0/180] [12769.784457] LNet: Added LNI 192.168.203.147@tcp110 [8/256/0/180] [12770.546163] LNet: Added LNI 192.168.203.147@tcp111 [8/256/0/180] [12771.248483] LNet: Added LNI 192.168.203.147@tcp112 [8/256/0/180] [12772.009493] LNet: Added LNI 192.168.203.147@tcp113 [8/256/0/180] [12772.749181] LNet: Added LNI 192.168.203.147@tcp114 [8/256/0/180] [12773.597915] LNet: Added LNI 192.168.203.147@tcp115 [8/256/0/180] [12774.376146] LNet: Added LNI 192.168.203.147@tcp116 [8/256/0/180] [12775.081707] LNet: Added LNI 192.168.203.147@tcp117 [8/256/0/180] [12775.742405] LNet: Added LNI 192.168.203.147@tcp118 [8/256/0/180] [12776.456750] LNet: Added LNI 192.168.203.147@tcp119 [8/256/0/180] [12777.216816] LNet: Added LNI 192.168.203.147@tcp120 [8/256/0/180] [12778.031520] LNet: Added LNI 192.168.203.147@tcp121 [8/256/0/180] [12778.838944] LNet: Added LNI 192.168.203.147@tcp122 [8/256/0/180] [12779.586894] LNet: Added LNI 192.168.203.147@tcp123 [8/256/0/180] [12780.511259] LNet: Added LNI 192.168.203.147@tcp124 [8/256/0/180] [12781.440252] LNet: Added LNI 192.168.203.147@tcp125 [8/256/0/180] [12782.299832] LNet: Added LNI 192.168.203.147@tcp126 [8/256/0/180] [12783.152994] LNet: Added LNI 192.168.203.147@tcp127 [8/256/0/180] [12784.054303] LNet: Added LNI 192.168.203.147@tcp128 [8/256/0/180] [12784.909070] LNet: Added LNI 192.168.203.147@tcp129 [8/256/0/180] [12785.790389] LNet: Added LNI 192.168.203.147@tcp130 [8/256/0/180] [12786.628666] LNet: Added LNI 192.168.203.147@tcp131 [8/256/0/180] [12787.474877] LNet: Added LNI 192.168.203.147@tcp132 [8/256/0/180] [12788.275114] LNet: Added LNI 192.168.203.147@tcp133 [8/256/0/180] [12789.127434] LNet: Added LNI 192.168.203.147@tcp134 [8/256/0/180] [12790.005484] LNet: Added LNI 192.168.203.147@tcp135 [8/256/0/180] [12796.947056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12796.982642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12798.167658] Lustre: 464288:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.147@tcp: can't add NID 192.168.203.147@tcp130: rc = -75 [12798.267175] LNetError: 464312:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12798.279757] LNetError: Protocol error connecting to 192.168.203.147@tcp111 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12798.286947] LNetError: Skipped 4 previous similar messages [12798.363069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12798.390567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12798.490637] Lustre: lustre-MDT0000: new disk, initializing [12798.562858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12798.591552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12799.137782] LNetError: 464312:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12799.139276] LNet: 4 local NIs in recovery (showing 4): 192.168.203.147@tcp124, 192.168.203.147@tcp125, 192.168.203.147@tcp126, 192.168.203.147@tcp127 [12799.148216] LNetError: 464312:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [12799.149304] LNetError: Protocol error connecting to 192.168.203.147@tcp112 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12799.149310] LNetError: Skipped 4 previous similar messages [12801.261767] LNetError: 464319:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12801.276550] LNetError: 464319:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [12801.289863] LNetError: Protocol error connecting to 192.168.203.147@tcp124 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12801.297645] LNetError: Skipped 5 previous similar messages [12802.926631] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12805.345820] LNetError: 464314:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12805.365623] LNetError: 464314:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [12805.370868] LNetError: Protocol error connecting to 192.168.203.147@tcp124 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12811.837399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12811.941170] Lustre: 464306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12811.997222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12812.000836] Lustre: Skipped 1 previous similar message [12812.049234] Lustre: lustre-MDT0001: new disk, initializing [12812.067848] Lustre: 465107:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.147@tcp: can't add NID 192.168.203.147@tcp130: rc = -75 [12812.073169] Lustre: 465107:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12812.149271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12812.189209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12812.205622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12813.541324] LNetError: 461738:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12813.548085] LNetError: Protocol error connecting to 192.168.203.147@tcp124 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12814.039437] Lustre: 465230:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.147@tcp: can't add NID 192.168.203.147@tcp130: rc = -75 [12814.044750] Lustre: 465230:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12815.539955] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12823.511388] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12828.836046] LNetError: 461737:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12828.846662] LNetError: Protocol error connecting to 192.168.203.147@tcp112 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12828.917696] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12835.736320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12836.079810] Lustre: lustre-OST0000: new disk, initializing [12836.086104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12836.090860] Lustre: 466282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12836.159985] Lustre: 466287:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.147@tcp: can't add NID 192.168.203.147@tcp130: rc = -75 [12836.165733] Lustre: 466287:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12836.220142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12841.773392] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12845.572182] Lustre: 466799:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.147@tcp: can't add NID 192.168.203.147@tcp130: rc = -75 [12845.583324] Lustre: 466799:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12845.625476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12845.637184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12845.748092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12848.290505] LNetError: 461736:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12848.305115] LNetError: 461736:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [12848.313158] LNetError: Protocol error connecting to 192.168.203.147@tcp129 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12848.327277] LNetError: Skipped 13 previous similar messages [12851.194627] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12859.615135] LNet: 1 local NIs in recovery (showing 1): 192.168.203.147@tcp124 [12861.418536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12861.418988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.429809] LustreError: Skipped 1 previous similar message [12861.441644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12861.456474] Lustre: Skipped 1 previous similar message [12863.627859] Lustre: server umount lustre-OST0000 complete [12868.583110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12868.591047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12868.599612] Lustre: Skipped 1 previous similar message [12868.612693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.654589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12871.656449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.696639] Lustre: Skipped 1 previous similar message [12876.769524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12876.777753] Lustre: Skipped 1 previous similar message [12881.375118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12881.583174] Lustre: server umount lustre-MDT0000 complete [12885.055037] LustreError: 464298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790110637 with bad export cookie 1414231282965413579 [12885.068799] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12885.071588] LustreError: 464298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12885.386783] Lustre: server umount lustre-MDT0001 complete [12893.347997] LNetError: 464316:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.147 [12893.354922] LNetError: 464316:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [12893.360937] LNetError: Protocol error connecting to 192.168.203.147@tcp120 at host 192.168.203.147:988: is it running a compatible version of Lustre? [12893.371250] LNetError: Skipped 17 previous similar messages [12894.868468] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12897.376477] Key type lgssc unregistered [12897.763885] LNet: 468194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12897.773844] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12897.783384] LNet: Removed LNI 192.168.203.147@tcp [12898.847191] LNet: Removed LNI 192.168.203.147@tcp100 [12898.852665] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12898.858728] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12898.864043] LNet: Removed LNI 192.168.203.147@tcp101 [12899.871590] LNet: Removed LNI 192.168.203.147@tcp102 [12899.876096] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12899.882657] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12899.899581] LNet: Removed LNI 192.168.203.147@tcp103 [12900.959146] LNet: Removed LNI 192.168.203.147@tcp104 [12901.984872] LNet: Removed LNI 192.168.203.147@tcp105 [12901.994452] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12902.003615] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12902.013539] LNet: Removed LNI 192.168.203.147@tcp106 [12903.071156] LNet: Removed LNI 192.168.203.147@tcp107 [12904.095204] LNet: Removed LNI 192.168.203.147@tcp108 [12905.119154] LNet: Removed LNI 192.168.203.147@tcp109 [12906.143117] LNet: Removed LNI 192.168.203.147@tcp110 [12906.146736] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12906.156616] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12906.164547] LNet: Removed LNI 192.168.203.147@tcp111 [12907.231121] LNet: Removed LNI 192.168.203.147@tcp112 [12908.255174] LNet: Removed LNI 192.168.203.147@tcp113 [12909.281475] LNet: Removed LNI 192.168.203.147@tcp114 [12910.304546] LNet: Removed LNI 192.168.203.147@tcp115 [12911.328942] LNet: Removed LNI 192.168.203.147@tcp116 [12912.351137] LNet: Removed LNI 192.168.203.147@tcp117 [12913.375674] LNet: Removed LNI 192.168.203.147@tcp118 [12914.399126] LNet: Removed LNI 192.168.203.147@tcp119 [12914.401210] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12914.419350] LNetError: 468194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12914.432533] LNet: Removed LNI 192.168.203.147@tcp120 [12915.488931] LNet: Removed LNI 192.168.203.147@tcp121 [12916.511138] LNet: Removed LNI 192.168.203.147@tcp122 [12917.535327] LNet: Removed LNI 192.168.203.147@tcp123 [12917.541279] LNet: Removed LNI 192.168.203.147@tcp124 [12918.559753] LNet: Removed LNI 192.168.203.147@tcp125 [12919.583143] LNet: Removed LNI 192.168.203.147@tcp126 [12920.609430] LNet: Removed LNI 192.168.203.147@tcp127 [12921.632169] LNet: Removed LNI 192.168.203.147@tcp128 [12922.655281] LNet: Removed LNI 192.168.203.147@tcp129 [12922.658186] LNet: Removed LNI 192.168.203.147@tcp130 [12922.660181] LNet: Removed LNI 192.168.203.147@tcp131 [12922.662269] LNet: Removed LNI 192.168.203.147@tcp132 [12922.664988] LNet: Removed LNI 192.168.203.147@tcp133 [12922.671394] LNet: Removed LNI 192.168.203.147@tcp134 [12922.687774] LNet: Removed LNI 192.168.203.147@tcp135 [12923.523779] Key type .llcrypt unregistered [12923.527744] Key type ._llcrypt unregistered [12941.895468] Key type ._llcrypt registered [12941.898617] Key type .llcrypt registered [12941.971712] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [12954.024160] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12955.021038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12955.041464] alg: No test for adler32 (adler32-zlib) [12956.077780] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [12956.280593] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12957.943452] Key type lgssc registered [12958.886655] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.283286] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:58:59 (1790110739) [13002.188318] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13004.401846] Key type lgssc unregistered [13004.647077] LNet: 472274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13004.658583] LNetError: 472274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13004.688524] LNet: Removed LNI 192.168.203.147@tcp [13005.275768] Key type .llcrypt unregistered [13005.280527] Key type ._llcrypt unregistered [13018.862800] Key type ._llcrypt registered [13018.865058] Key type .llcrypt registered [13018.969938] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13020.013745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13020.063241] alg: No test for adler32 (adler32-zlib) [13021.191584] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [13021.521595] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13023.210721] Key type lgssc registered [13024.389759] Lustre: Echo OBD driver; http://www.lustre.org/ [13032.556268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13032.585135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13033.792746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13033.863901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13033.917860] Lustre: lustre-MDT0000: new disk, initializing [13033.963207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13033.973685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13038.508782] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13047.473703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13047.571905] Lustre: 473482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13047.629675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13047.633458] Lustre: Skipped 1 previous similar message [13047.698549] Lustre: lustre-MDT0001: new disk, initializing [13047.779552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13047.825311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13047.837898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13051.216924] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13058.783597] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13063.177459] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13068.360370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13068.585518] Lustre: lustre-OST0000: new disk, initializing [13068.592751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13068.601549] Lustre: 475447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13068.696606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13073.496136] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13075.502647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13075.509857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13075.591319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13081.157639] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13084.216580] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing lnet_if_list [13088.231046] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13088.262045] LNet: Added LNI 192.168.203.147@tcp10 [8/256/0/180] [13092.707373] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13092.732138] LNet: Added LNI 192.168.203.147@tcp11 [8/256/0/180] [13096.103729] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13096.127675] LNet: Added LNI 192.168.203.147@tcp12 [8/256/0/180] [13099.550332] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13099.577992] LNet: Added LNI 192.168.203.147@tcp13 [8/256/0/180] [13102.480722] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13102.507382] LNet: Added LNI 192.168.203.147@tcp14 [8/256/0/180] [13105.819927] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13105.841650] LNet: Added LNI 192.168.203.147@tcp15 [8/256/0/180] [13109.482917] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13109.512940] LNet: Added LNI 192.168.203.147@tcp16 [8/256/0/180] [13113.706212] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13113.727440] LNet: Added LNI 192.168.203.147@tcp17 [8/256/0/180] [13117.229071] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13117.250775] LNet: Added LNI 192.168.203.147@tcp18 [8/256/0/180] [13120.953471] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13120.975100] LNet: Added LNI 192.168.203.147@tcp19 [8/256/0/180] [13125.091172] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13125.126864] LNet: Added LNI 192.168.203.147@tcp20 [8/256/0/180] [13129.404881] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13129.424775] LNet: Added LNI 192.168.203.147@tcp21 [8/256/0/180] [13133.716663] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13133.746571] LNet: Added LNI 192.168.203.147@tcp22 [8/256/0/180] [13138.125881] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13138.154367] LNet: Added LNI 192.168.203.147@tcp23 [8/256/0/180] [13142.166708] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13142.187403] LNet: Added LNI 192.168.203.147@tcp24 [8/256/0/180] [13146.241355] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13146.274622] LNet: Added LNI 192.168.203.147@tcp25 [8/256/0/180] [13150.638391] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13150.671120] LNet: Added LNI 192.168.203.147@tcp26 [8/256/0/180] [13154.634583] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13154.656979] LNet: Added LNI 192.168.203.147@tcp27 [8/256/0/180] [13159.201368] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13159.234242] LNet: Added LNI 192.168.203.147@tcp28 [8/256/0/180] [13163.881584] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13163.913763] LNet: Added LNI 192.168.203.147@tcp29 [8/256/0/180] [13168.135492] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13168.158656] LNet: Added LNI 192.168.203.147@tcp30 [8/256/0/180] [13172.520812] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13172.549525] LNet: Added LNI 192.168.203.147@tcp31 [8/256/0/180] [13177.154919] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13177.189235] LNet: Added LNI 192.168.203.147@tcp32 [8/256/0/180] [13182.103865] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13182.130035] LNet: Added LNI 192.168.203.147@tcp33 [8/256/0/180] [13186.828980] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13186.848346] LNet: Added LNI 192.168.203.147@tcp34 [8/256/0/180] [13192.104968] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13192.131892] LNet: Added LNI 192.168.203.147@tcp35 [8/256/0/180] [13197.226743] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13197.258379] LNet: Added LNI 192.168.203.147@tcp36 [8/256/0/180] [13202.557298] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13202.579129] LNet: Added LNI 192.168.203.147@tcp37 [8/256/0/180] [13208.215284] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13208.244705] LNet: Added LNI 192.168.203.147@tcp38 [8/256/0/180] [13213.610859] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13213.631514] LNet: Added LNI 192.168.203.147@tcp39 [8/256/0/180] [13219.219283] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13219.247858] LNet: Added LNI 192.168.203.147@tcp40 [8/256/0/180] [13224.672742] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13224.692150] LNet: Added LNI 192.168.203.147@tcp41 [8/256/0/180] [13230.227661] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13230.266098] LNet: Added LNI 192.168.203.147@tcp42 [8/256/0/180] [13235.823500] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13235.842882] LNet: Added LNI 192.168.203.147@tcp43 [8/256/0/180] [13241.203085] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13241.226520] LNet: Added LNI 192.168.203.147@tcp44 [8/256/0/180] [13247.275674] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13247.328602] LNet: Added LNI 192.168.203.147@tcp45 [8/256/0/180] [13253.622562] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13253.652127] LNet: Added LNI 192.168.203.147@tcp46 [8/256/0/180] [13259.394655] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13259.418462] LNet: Added LNI 192.168.203.147@tcp47 [8/256/0/180] [13265.450436] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13265.477238] LNet: Added LNI 192.168.203.147@tcp48 [8/256/0/180] [13271.521048] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13271.564848] LNet: Added LNI 192.168.203.147@tcp49 [8/256/0/180] [13277.407862] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13277.433900] LNet: Added LNI 192.168.203.147@tcp50 [8/256/0/180] [13290.467716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13290.469763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13290.479611] Lustre: Skipped 2 previous similar messages [13290.489852] Lustre: Skipped 2 previous similar messages [13291.636193] Lustre: server umount lustre-MDT0000 complete [13294.156884] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111046 with bad export cookie 13596916793162243935 [13294.163904] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13294.168617] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13294.432853] Lustre: server umount lustre-MDT0001 complete [13308.223485] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13311.905076] Lustre: 472746:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790111047/real 1790111047] req@00000000e73ca6c3 x1877067194043904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790111063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13311.921411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13314.233612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13315.999182] Lustre: 472744:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790111051/real 1790111051] req@00000000a0661f3e x1877067194044032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790111067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13317.087138] Lustre: 472744:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790111052/real 1790111052] req@00000000a2f938ce x1877067194044160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790111068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13319.135778] LustreError: 472742:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bff810dd x1877067194045056/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13319.390119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13320.415433] Lustre: 472746:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790111056/real 1790111056] req@0000000017c6a871 x1877067194044416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790111072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13322.317951] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13325.539771] Lustre: 472745:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790111061/real 1790111061] req@00000000dd3b1413 x1877067194044544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790111077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13326.820847] LustreError: 499660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.833068] LustreError: 499660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13327.688283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13327.949266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13330.954703] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13331.048700] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13335.924516] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13340.027988] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13353.440882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13353.449677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13353.469571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13355.362129] Lustre: server umount lustre-OST0000 complete [13358.564355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13358.571451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13358.587050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13362.145153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13362.152231] Lustre: Skipped 1 previous similar message [13367.265375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13367.274273] Lustre: Skipped 1 previous similar message [13368.354347] Lustre: server umount lustre-MDT0000 complete [13370.892895] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111122 with bad export cookie 13596916793162246378 [13370.897668] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13370.908526] LustreError: 473472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13371.112971] Lustre: server umount lustre-MDT0001 complete [13378.452078] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13380.261749] Key type lgssc unregistered [13380.538115] LNet: 502435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13380.545824] LNetError: 502435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13381.599410] LNet: Removed LNI 192.168.203.147@tcp [13381.602460] LNetError: 502435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13381.606260] LNet: Removed LNI 192.168.203.147@tcp10 [13381.610311] LNet: Removed LNI 192.168.203.147@tcp11 [13381.614636] LNet: Removed LNI 192.168.203.147@tcp12 [13381.620368] LNet: Removed LNI 192.168.203.147@tcp13 [13381.623393] LNet: Removed LNI 192.168.203.147@tcp14 [13381.627525] LNet: Removed LNI 192.168.203.147@tcp15 [13381.631963] LNet: Removed LNI 192.168.203.147@tcp16 [13381.634213] LNet: Removed LNI 192.168.203.147@tcp17 [13381.636440] LNet: Removed LNI 192.168.203.147@tcp18 [13381.639104] LNet: Removed LNI 192.168.203.147@tcp19 [13381.641617] LNet: Removed LNI 192.168.203.147@tcp20 [13381.643823] LNet: Removed LNI 192.168.203.147@tcp21 [13381.646365] LNet: Removed LNI 192.168.203.147@tcp22 [13381.648668] LNet: Removed LNI 192.168.203.147@tcp23 [13381.651371] LNet: Removed LNI 192.168.203.147@tcp24 [13381.653646] LNet: Removed LNI 192.168.203.147@tcp25 [13381.656119] LNet: Removed LNI 192.168.203.147@tcp26 [13381.658362] LNet: Removed LNI 192.168.203.147@tcp27 [13381.660531] LNet: Removed LNI 192.168.203.147@tcp28 [13381.663033] LNet: Removed LNI 192.168.203.147@tcp29 [13381.664744] LNet: Removed LNI 192.168.203.147@tcp30 [13381.666734] LNet: Removed LNI 192.168.203.147@tcp31 [13381.670391] LNet: Removed LNI 192.168.203.147@tcp32 [13381.672585] LNet: Removed LNI 192.168.203.147@tcp33 [13381.677951] LNet: Removed LNI 192.168.203.147@tcp34 [13381.680469] LNet: Removed LNI 192.168.203.147@tcp35 [13381.682823] LNet: Removed LNI 192.168.203.147@tcp36 [13381.690954] LNet: Removed LNI 192.168.203.147@tcp37 [13381.698329] LNet: Removed LNI 192.168.203.147@tcp38 [13381.703633] LNet: Removed LNI 192.168.203.147@tcp39 [13381.705789] LNet: Removed LNI 192.168.203.147@tcp40 [13381.708209] LNet: Removed LNI 192.168.203.147@tcp41 [13381.712421] LNet: Removed LNI 192.168.203.147@tcp42 [13381.719953] LNet: Removed LNI 192.168.203.147@tcp43 [13381.721742] LNet: Removed LNI 192.168.203.147@tcp44 [13381.725436] LNet: Removed LNI 192.168.203.147@tcp45 [13381.727365] LNet: Removed LNI 192.168.203.147@tcp46 [13381.730123] LNet: Removed LNI 192.168.203.147@tcp47 [13381.733891] LNet: Removed LNI 192.168.203.147@tcp48 [13381.735812] LNet: Removed LNI 192.168.203.147@tcp49 [13381.744821] LNet: Removed LNI 192.168.203.147@tcp50 [13382.289275] Key type .llcrypt unregistered [13382.291494] Key type ._llcrypt unregistered [13396.673959] Key type ._llcrypt registered [13396.676274] Key type .llcrypt registered [13396.736881] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [13406.240533] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13406.826530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13406.860337] alg: No test for adler32 (adler32-zlib) [13407.814830] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [13407.982701] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13409.607240] Key type lgssc registered [13410.379139] Lustre: Echo OBD driver; http://www.lustre.org/ [13432.829845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13432.860880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13434.017614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13434.040359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13434.097750] Lustre: lustre-MDT0000: new disk, initializing [13434.161621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13434.178768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13436.998493] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13443.883283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13443.963582] Lustre: 505623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13443.993201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13443.995561] Lustre: Skipped 1 previous similar message [13444.046907] Lustre: lustre-MDT0001: new disk, initializing [13444.102536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13444.136515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13444.153811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13446.759992] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13453.259812] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13457.291492] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13462.667922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13462.828250] Lustre: *** cfs_fail_loc=90f, val=0*** [13462.886192] Lustre: lustre-OST0000: new disk, initializing [13462.888751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13462.892763] Lustre: 507638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13462.952978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13466.846608] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13473.314073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13473.326666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13473.425070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13473.737394] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13483.488680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13483.493264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13483.501670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13484.001993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13484.012333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13484.024410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13487.011276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13487.015021] Lustre: Skipped 1 previous similar message [13492.198022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13492.207029] Lustre: Skipped 1 previous similar message [13493.215665] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13493.349677] Lustre: server umount lustre-OST0000 complete [13497.313765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13497.316060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13497.323582] Lustre: Skipped 1 previous similar message [13497.329350] Lustre: Skipped 1 previous similar message [13501.679944] Lustre: server umount lustre-MDT0000 complete [13503.837217] LustreError: 505618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111255 with bad export cookie 4141024978607354571 [13503.838802] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13503.841510] LustreError: 505618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13503.980937] Lustre: server umount lustre-MDT0001 complete [13509.205732] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13510.575819] Key type lgssc unregistered [13510.730658] LNet: 509543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13510.737585] LNetError: 509543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13510.751560] LNet: Removed LNI 192.168.203.147@tcp [13511.160798] Key type .llcrypt unregistered [13511.162620] Key type ._llcrypt unregistered [13522.940624] Key type ._llcrypt registered [13522.943623] Key type .llcrypt registered [13523.014430] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13523.488912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13523.499832] alg: No test for adler32 (adler32-zlib) [13524.456627] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [13524.615887] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13526.239252] Key type lgssc registered [13526.994553] Lustre: Echo OBD driver; http://www.lustre.org/ [13531.558258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13531.569969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13532.825818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13534.950635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13540.033033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13540.252382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13542.736545] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13547.864949] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13551.957349] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13556.919236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13557.160421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13561.078853] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13565.786976] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13573.014657] loop0: detected capacity change from 0 to 204800000 [13573.153895] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13573.163897] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13573.172254] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13576.933919] loop0: detected capacity change from 0 to 204800000 [13577.038307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13577.049097] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13577.059612] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13580.638217] loop0: detected capacity change from 0 to 204800000 [13583.993120] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13584.089756] Lustre: fs15246-OST0000: new disk, initializing [13584.092625] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13584.096631] Lustre: 514396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13584.135356] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13588.087228] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13595.043884] loop1: detected capacity change from 0 to 204800000 [13599.098337] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13599.207839] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13599.270177] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13599.322255] Lustre: fs15246-MDT0000: new disk, initializing [13599.448134] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13599.473771] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13599.555912] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13600.610455] Lustre: 515517:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005f195563 x1877067721381376/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790111368 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13602.569834] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13609.700862] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13609.706841] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13609.723962] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13618.658635] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13618.669879] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13623.306171] Lustre: server umount fs15246-MDT0000 complete [13623.783357] LustreError: 510802:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13623.800577] LustreError: 510802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13628.915892] LustreError: 510802:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13629.879314] Lustre: server umount fs15246-OST0000 complete [13639.142061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13639.146185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13639.153451] Lustre: Skipped 1 previous similar message [13639.166793] Lustre: Skipped 1 previous similar message [13644.259713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13644.267242] Lustre: Skipped 1 previous similar message [13644.881511] Lustre: server umount lustre-OST0000 complete [13647.840078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13647.845347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13647.853096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13654.496807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13654.499494] Lustre: Skipped 2 previous similar messages [13662.175155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13662.274745] Lustre: server umount lustre-MDT0000 complete [13664.744752] LustreError: 511729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13664.764113] LustreError: 511729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13664.811636] LustreError: 510787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111416 with bad export cookie 15560643864455675164 [13664.816373] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13664.817730] LustreError: 510787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13665.065937] Lustre: server umount lustre-MDT0001 complete [13679.587722] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13683.675857] Key type lgssc unregistered [13684.126832] LNet: 518613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13684.137831] LNetError: 518613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13684.165225] LNet: Removed LNI 192.168.203.147@tcp [13685.977181] Key type .llcrypt unregistered [13685.979903] Key type ._llcrypt unregistered [13711.396684] Key type ._llcrypt registered [13711.399933] Key type .llcrypt registered [13711.513307] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13713.123681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13713.155266] alg: No test for adler32 (adler32-zlib) [13714.210971] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [13714.476168] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13716.143211] Key type lgssc registered [13717.572888] Lustre: Echo OBD driver; http://www.lustre.org/ [13764.710253] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [13773.636827] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13804.944749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13804.963857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13806.218903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13806.267916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13806.356343] Lustre: lustre-MDT0000: new disk, initializing [13806.443487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13806.470211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13810.736222] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13820.438731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13820.587629] Lustre: 523096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13820.666407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13820.676965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13820.687923] Lustre: Skipped 1 previous similar message [13820.787391] Lustre: lustre-MDT0001: new disk, initializing [13820.911798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13820.988703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13821.000425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13825.131045] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13834.557334] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13840.395502] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13847.735085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13848.063533] Lustre: lustre-OST0000: new disk, initializing [13848.068772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13848.077698] Lustre: 525066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13848.171623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13854.816915] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13857.900454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13857.915943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13858.128940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13865.006471] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13870.162267] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13870.433296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13875.352330] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13875.583297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13878.770401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13878.783661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13878.798766] Lustre: Skipped 1 previous similar message [13878.800770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13878.807340] Lustre: Skipped 1 previous similar message [13878.832728] LustreError: Skipped 1 previous similar message [13883.790748] Lustre: server umount lustre-OST0000 complete [13888.993230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13889.014728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13889.020611] Lustre: Skipped 1 previous similar message [13893.929552] Lustre: server umount lustre-MDT0000 complete [13897.747691] LustreError: 525065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790111649 with bad export cookie 733166753886745856 [13897.750549] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13897.765857] LustreError: 525065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13898.257663] Lustre: server umount lustre-MDT0001 complete [13923.736337] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [13934.389717] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13966.060183] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13734 sec ========= 17:15:16 (1790111716) [13967.885469] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:15:18 (1790111718) === [13971.628424] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:15:22 (1790111722) === [14012.368232] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [14015.183625] Key type lgssc unregistered [14015.448949] LNet: 531710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14015.458926] LNetError: 531710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14015.476113] LNet: Removed LNI 192.168.203.147@tcp [14016.436870] Key type .llcrypt unregistered [14016.442084] Key type ._llcrypt unregistered