[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467760161 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003085] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008285] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011129] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.015018] *** VALIDATE selinux *** [ 0.023676] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028608] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032106] *** VALIDATE tmpfs *** [ 0.033476] *** VALIDATE proc *** [ 0.034248] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044016] debug: unmapping init [mem 0xffffffff84e59000-0xffffffff84e60fff] [ 0.046178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054019] ... event mask: 000000070000000f [ 0.055328] rcu: Hierarchical SRCU implementation. [ 0.057429] smp: Bringing up secondary CPUs ... [ 0.058569] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.062482] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138434] node 0 deferred pages initialised in 71ms [ 0.142008] devtmpfs: initialized [ 0.143266] x86/mm: Memory block size: 128MB [ 0.144000] gcov: version magic: 0x41383552 [ 0.147237] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151121] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153400] pinctrl core: initialized pinctrl subsystem [ 0.156174] [ 0.156650] ************************************************************* [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161013] ** ** [ 0.163015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166012] ** ** [ 0.168012] ** This means that this kernel is built to expose internal ** [ 0.170012] ** IOMMU data structures, which may compromise security on ** [ 0.171009] ** your system. ** [ 0.174012] ** ** [ 0.176012] ** If you see this message and you are not debugging the ** [ 0.178028] ** kernel, report this immediately to your vendor! ** [ 0.180013] ** ** [ 0.182012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184024] ************************************************************* [ 0.187671] NET: Registered protocol family 16 [ 0.189414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196587] cpuidle: using governor menu [ 0.198000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201525] PCI: Using configuration type 1 for base access [ 0.204131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218182] cryptd: max_cpu_qlen set to 1000 [ 0.220453] ACPI: Added _OSI(Module Device) [ 0.221016] ACPI: Added _OSI(Processor Device) [ 0.222014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233009] ACPI: Interpreter enabled [ 0.234047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235011] ACPI: Using IOAPIC for interrupt routing [ 0.236099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248449] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260464] acpiphp: Slot [2] registered [ 0.262179] acpiphp: Slot [5] registered [ 0.264119] acpiphp: Slot [6] registered [ 0.266107] acpiphp: Slot [7] registered [ 0.267129] acpiphp: Slot [8] registered [ 0.269099] acpiphp: Slot [9] registered [ 0.271080] acpiphp: Slot [10] registered [ 0.273089] acpiphp: Slot [3] registered [ 0.275199] acpiphp: Slot [4] registered [ 0.277095] acpiphp: Slot [11] registered [ 0.279096] acpiphp: Slot [12] registered [ 0.281098] acpiphp: Slot [13] registered [ 0.282109] acpiphp: Slot [14] registered [ 0.284161] acpiphp: Slot [15] registered [ 0.286131] acpiphp: Slot [16] registered [ 0.288084] acpiphp: Slot [17] registered [ 0.289090] acpiphp: Slot [18] registered [ 0.290081] acpiphp: Slot [19] registered [ 0.292095] acpiphp: Slot [20] registered [ 0.294107] acpiphp: Slot [21] registered [ 0.295100] acpiphp: Slot [22] registered [ 0.297077] acpiphp: Slot [23] registered [ 0.298115] acpiphp: Slot [24] registered [ 0.300138] acpiphp: Slot [25] registered [ 0.302126] acpiphp: Slot [26] registered [ 0.304121] acpiphp: Slot [27] registered [ 0.306088] acpiphp: Slot [28] registered [ 0.307095] acpiphp: Slot [29] registered [ 0.309085] acpiphp: Slot [30] registered [ 0.310089] acpiphp: Slot [31] registered [ 0.312062] PCI host bridge to bus 0000:00 [ 0.313029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327169] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340792] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348088] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355063] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357985] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365178] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395755] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.501190] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520072] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553122] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599469] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.642191] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.655019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689778] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702334] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.707157] iommu: Default domain type: Passthrough [ 0.709409] SCSI subsystem initialized [ 0.711135] ACPI: bus type USB registered [ 0.712100] usbcore: registered new interface driver usbfs [ 0.716081] usbcore: registered new interface driver hub [ 0.717062] usbcore: registered new device driver usb [ 0.719230] pps_core: LinuxPPS API ver. 1 registered [ 0.721018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.724084] PTP clock support registered [ 0.726077] EDAC MC: Ver: 3.0.0 [ 0.728088] PCI: Using ACPI for IRQ routing [ 0.729576] NetLabel: Initializing [ 0.731019] NetLabel: domain hash size = 128 [ 0.732011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.735123] NetLabel: unlabeled traffic allowed by default [ 0.737171] vgaarb: loaded [ 0.739350] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.741014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744643] clocksource: Switched to clocksource kvm-clock [ 0.865651] VFS: Disk quotas dquot_6.6.0 [ 0.867135] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.869603] *** VALIDATE ramfs *** [ 0.870845] *** VALIDATE hugetlbfs *** [ 0.872519] pnp: PnP ACPI init [ 0.874593] pnp: PnP ACPI: found 6 devices [ 0.893433] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.897752] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.900481] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.902995] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.905788] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.908889] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.913194] NET: Registered protocol family 2 [ 0.916209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921738] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.925827] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.931610] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.935389] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937635] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.940751] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943918] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947182] NET: Registered protocol family 1 [ 0.950409] RPC: Registered named UNIX socket transport module. [ 0.952789] RPC: Registered udp transport module. [ 0.954506] RPC: Registered tcp transport module. [ 0.956495] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.959128] NET: Registered protocol family 44 [ 0.961203] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.963670] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966117] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968696] PCI: CLS 0 bytes, default 64 [ 0.970600] Unpacking initramfs... [ 2.375545] debug: unmapping init [mem 0xffff8cfefcc54000-0xffff8cfefffbffff] [ 2.380974] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.383356] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.386411] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926045] Initialise system trusted keyrings [ 2.928433] Key type blacklist registered [ 2.930395] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.942208] zbud: loaded [ 2.945476] *** VALIDATE nfs *** [ 2.946775] *** VALIDATE nfs4 *** [ 2.948298] pstore: using deflate compression [ 2.952054] Platform Keyring initialized [ 3.060464] NET: Registered protocol family 38 [ 3.062392] Key type asymmetric registered [ 3.063977] Asymmetric key parser 'x509' registered [ 3.065942] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.070372] io scheduler mq-deadline registered [ 3.072076] io scheduler kyber registered [ 3.073227] io scheduler bfq registered [ 3.075229] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.078287] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.081305] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.084289] ACPI: Power Button [PWRF] [ 3.089660] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.097353] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.121083] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.132114] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.156054] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.183200] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.213486] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.219562] Non-volatile memory driver v1.3 [ 3.221873] Linux agpgart interface v0.103 [ 3.262289] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.267872] vda: detected capacity change from 0 to 74690560 [ 3.287394] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.290791] vdb: detected capacity change from 0 to 1073741824 [ 3.304710] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308148] vdc: detected capacity change from 0 to 2621440000 [ 3.325477] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.329155] vdd: detected capacity change from 0 to 2621440000 [ 3.346222] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.348747] vde: detected capacity change from 0 to 4294967296 [ 3.368071] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.371942] vdf: detected capacity change from 0 to 4294967296 [ 3.378993] libphy: Fixed MDIO Bus: probed [ 3.384572] usbcore: registered new interface driver usbserial_generic [ 3.387340] usbserial: USB Serial support registered for generic [ 3.389768] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.393541] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.395411] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.398215] mousedev: PS/2 mouse device common for all mice [ 3.401632] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.405596] rtc_cmos 00:05: RTC can wake from S4 [ 3.409382] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.414591] rtc_cmos 00:05: registered as rtc0 [ 3.418447] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.418963] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.423184] intel_pstate: CPU model not supported [ 3.430462] hid: raw HID events driver (C) Jiri Kosina [ 3.432768] usbcore: registered new interface driver usbhid [ 3.434898] usbhid: USB HID core driver [ 3.436481] drop_monitor: Initializing network drop monitor service [ 3.438955] Initializing XFRM netlink socket [ 3.441642] NET: Registered protocol family 10 [ 3.445589] Segment Routing with IPv6 [ 3.446953] NET: Registered protocol family 17 [ 3.451245] mpls_gso: MPLS GSO support [ 3.457939] RAS: Correctable Errors collector initialized. [ 3.459898] AVX version of gcm_enc/dec engaged. [ 3.461432] AES CTR mode by8 optimization enabled [ 3.540710] sched_clock: Marking stable (3540686701, 0)->(4457554310, -916867609) [ 3.543624] registered taskstats version 1 [ 3.545355] Loading compiled-in X.509 certificates [ 3.547180] zswap: loaded using pool lzo/zbud [ 3.570266] Key type big_key registered [ 3.582764] Key type encrypted registered [ 3.584259] ima: No TPM chip found, activating TPM-bypass! [ 3.585890] ima: Allocated hash algorithm: sha1 [ 3.587233] ima: No architecture policies found [ 3.588491] evm: Initialising EVM extended attributes: [ 3.589892] evm: security.selinux [ 3.591209] evm: security.ima [ 3.591973] evm: security.capability [ 3.593009] evm: HMAC attrs: 0x1 [ 3.595093] rtc_cmos 00:05: setting system clock to 2026-07-29 07:24:56 UTC (1785309896) [ 3.601299] debug: unmapping init [mem 0xffffffff85e03000-0xffffffff85ffffff] [ 3.603883] debug: unmapping init [mem 0xffffffff84b82000-0xffffffff84e58fff] [ 3.607601] Write protecting the kernel read-only data: 28672k [ 3.610673] debug: unmapping init [mem 0xffffffff83203000-0xffffffff833fffff] [ 3.613030] debug: unmapping init [mem 0xffffffff83b14000-0xffffffff83bfffff] [ 3.650622] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.657960] systemd[1]: Detected virtualization kvm. [ 3.660579] systemd[1]: Detected architecture x86-64. [ 3.662912] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.690849] systemd[1]: No hostname configured. [ 3.692867] systemd[1]: Set hostname to . [ 3.694796] random: systemd: uninitialized urandom read (16 bytes read) [ 3.696929] systemd[1]: Initializing machine ID from random generator. [ 3.863373] random: systemd: uninitialized urandom read (16 bytes read) [ 3.866342] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.872531] random: systemd: uninitialized urandom read (16 bytes read) [ 3.874671] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.879514] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.584989] device-mapper: uevent: version 1.0.3 [ 4.587361] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.297081] virtio_net virtio0 ens2: renamed from eth0 [ 5.315891] scsi host0: ata_piix [ 5.328766] scsi host1: ata_piix [ 5.330482] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.334033] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.353563] random: fast init done [ 9.493848] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.109116] random: crng init done [ 10.110378] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.439976] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.521790] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.801508] SELinux: Disabled at runtime. [ 11.861869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.871597] systemd[1]: Detected virtualization kvm. [ 11.873119] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.361527] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.364577] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.369314] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.374230] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.377661] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.385478] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.392995] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.486994] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.797136] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.137783] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.175291] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.301567] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.323284] EDAC sbridge: Ver: 1.1.2 [ 15.349615] Key type dns_resolver registered [ 15.659959] NFS: Registering the id_resolver key type [ 15.662045] Key type id_resolver registered [ 15.663615] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg615-server login: [ 45.172751] libcfs: loading out-of-tree module taints kernel. [ 45.220740] Key type ._llcrypt registered [ 45.222585] Key type .llcrypt registered [ 45.309469] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 62.757401] hrtimer: interrupt took 3349823 ns [ 67.110486] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 69.753724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.783027] alg: No test for adler32 (adler32-zlib) [ 71.433813] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 72.409610] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 74.144694] Key type lgssc registered [ 75.831530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.526779] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.422673] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 142.857280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.894035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.165678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.190577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.302603] Lustre: lustre-MDT0000: new disk, initializing [ 144.450642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.469683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.991467] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.207571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.366024] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.438888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.444533] Lustre: Skipped 1 previous similar message [ 164.527980] Lustre: lustre-MDT0001: new disk, initializing [ 164.663086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.704344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.716858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.449267] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.350685] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.099654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.326412] Lustre: lustre-OST0000: new disk, initializing [ 185.336073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.348701] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.461189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.024896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.036322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.175289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.509837] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.139698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.267793] Lustre: lustre-OST0001: new disk, initializing [ 209.274262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.283371] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.360966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.155051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.166602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.253700] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.454258] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.322937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.190216] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.939973] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.787181] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing yml_node [ 251.978523] Lustre: DEBUG MARKER: Client: 2.17.55.22 [ 255.259169] Lustre: DEBUG MARKER: MDS: 2.17.55.22 [ 257.508409] Lustre: DEBUG MARKER: OSS: 2.17.55.22 [ 259.026768] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 29 03:29:10 EDT 2026 [ 275.883829] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 277.312434] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 292.834242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 292.840812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.848101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.344989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.368824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.377134] Lustre: Skipped 3 previous similar messages [ 298.465541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.482079] Lustre: Skipped 3 previous similar messages [ 303.589270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.596031] Lustre: Skipped 3 previous similar messages [ 306.144537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 306.362948] Lustre: server umount lustre-MDT0000 complete [ 309.688048] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310202 with bad export cookie 8167060883473919575 [ 309.694688] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.704274] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 313.829260] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.829630] LustreError: 12148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.830210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.851570] Lustre: Skipped 3 previous similar messages [ 313.893210] LustreError: 12148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 314.849913] LustreError: 8255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.861188] LustreError: 8255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 316.138911] Lustre: server umount lustre-MDT0001 complete [ 319.821789] Lustre: server umount lustre-OST0000 complete [ 323.611632] Lustre: server umount lustre-OST0001 complete [ 330.124238] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 338.388966] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 366.014866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.219814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 366.249958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 366.338381] Lustre: lustre-MDT0000: new disk, initializing [ 366.395331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.406831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 370.050386] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 379.011568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.154854] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.257951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.263767] Lustre: Skipped 1 previous similar message [ 379.369429] Lustre: lustre-MDT0001: new disk, initializing [ 379.557857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.624543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.639088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.646614] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 398.411669] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 405.424873] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 412.437520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.773189] Lustre: lustre-OST0000: new disk, initializing [ 412.776690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 412.783755] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 412.898808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.692049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 414.707174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 414.875455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.678808] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 429.629276] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 434.075375] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 434.333761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 437.726291] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 437.947526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.294656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.304939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.305928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.309030] Lustre: Skipped 3 previous similar messages [ 445.768876] Lustre: server umount lustre-OST0000 complete [ 450.528972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.547790] Lustre: Skipped 1 previous similar message [ 455.373849] Lustre: server umount lustre-MDT0000 complete [ 455.650922] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.670196] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 458.634396] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310351 with bad export cookie 8167060883473921108 [ 458.635900] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.650557] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 458.981624] Lustre: server umount lustre-MDT0001 complete [ 463.116758] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:32:34 (1785310354) [ 465.256432] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 466.600263] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:32:38 (1785310358) [ 472.392041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.919155] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.995632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.063180] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 478.176917] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.120441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.591939] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 495.557970] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.977283] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.113058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 513.308513] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 520.007892] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 527.841201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.843392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.849248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.849255] Lustre: Skipped 5 previous similar messages [ 527.862765] Lustre: Skipped 2 previous similar messages [ 532.309705] Lustre: server umount lustre-OST0000 complete [ 536.546477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.367451] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.545839] Lustre: server umount lustre-MDT0000 complete [ 552.417870] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.432228] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 553.780579] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310446 with bad export cookie 8167060883473921990 [ 553.786414] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.806081] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 554.086971] Lustre: server umount lustre-MDT0001 complete [ 563.453340] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 565.975867] Key type lgssc unregistered [ 566.235814] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.242296] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.254983] LNet: Removed LNI 192.168.206.115@tcp [ 567.377937] Key type .llcrypt unregistered [ 567.380793] Key type ._llcrypt unregistered [ 586.548863] Key type ._llcrypt registered [ 586.550510] Key type .llcrypt registered [ 586.662551] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 587.779333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 587.829595] alg: No test for adler32 (adler32-zlib) [ 588.786828] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 589.112957] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 590.879240] Key type lgssc registered [ 591.986894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.530416] loop: module loaded [ 601.613993] loop0: detected capacity change from 0 to 204800000 [ 601.856700] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 601.880617] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.198643] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.570481] loop0: detected capacity change from 0 to 204800000 [ 608.760819] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.779962] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.791601] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.210675] loop0: detected capacity change from 0 to 204800000 [ 618.644816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 618.683903] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.043240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 620.114772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 620.294723] Lustre: lustre-MDT0000: new disk, initializing [ 620.526045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.573219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 625.138546] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 634.294745] loop1: detected capacity change from 0 to 204800000 [ 639.799776] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 640.098315] Lustre: lustre-OST07c6: new disk, initializing [ 640.102056] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 640.106790] Lustre: Skipped 1 previous similar message [ 640.111380] Lustre: 26148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 640.168826] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 645.659446] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 647.753552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 647.769626] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 647.804742] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 649.774101] Lustre: 25133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 649.782562] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 649.786819] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 649.796264] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 649.800952] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 649.806268] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 668.449827] Lustre: 25133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 668.457383] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 668.463231] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.468082] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 668.474163] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 668.482633] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.319615] Lustre: 25133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 673.333719] Lustre: 25133:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.340835] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 673.345865] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.350841] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.360290] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.369671] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 673.382049] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.392371] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.397423] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.403565] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.408414] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.585535] Lustre: 26156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 676.610757] Lustre: 26156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 676.622251] Lustre: 26156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 676.634615] Lustre: 26156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.656723] Lustre: 26156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.667737] Lustre: 26156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.678249] Lustre: 26156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 676.684393] Lustre: 26156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.691452] Lustre: 26156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.700784] Lustre: 26156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.713430] Lustre: 26156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.721141] Lustre: 26156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.409815] Lustre: 25134:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 681.423222] Lustre: 25134:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 681.429418] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 681.438050] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.455401] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.463807] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.471785] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 681.487376] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.493596] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 681.498648] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.506219] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 681.519763] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.763536] Lustre: 25134:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 692.780607] Lustre: 25134:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 692.795257] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 692.802087] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.810356] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 692.818724] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.829224] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 692.836613] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.844168] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 692.854324] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.862880] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.869717] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.929371] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 744.941845] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.961518] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 747.587308] Lustre: server umount lustre-OST07c6 complete [ 756.479820] Lustre: server umount lustre-MDT0000 complete [ 773.112556] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 776.198227] Key type lgssc unregistered [ 776.496883] LNet: 28705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.505419] LNetError: 28705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.532429] LNet: Removed LNI 192.168.206.115@tcp [ 777.384711] Key type .llcrypt unregistered [ 777.387160] Key type ._llcrypt unregistered [ 796.447655] Key type ._llcrypt registered [ 796.451448] Key type .llcrypt registered [ 796.565941] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 797.890843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.996120] alg: No test for adler32 (adler32-zlib) [ 799.152414] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 799.618624] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 801.351580] Key type lgssc registered [ 802.741743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 810.749987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 810.772216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.200235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.448754] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 822.762371] LustreError: 29976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.778921] LustreError: 29976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 827.415393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.910290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.899616] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 841.295607] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 847.747834] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 855.069374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 855.503991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.619157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 862.332927] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 870.208713] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.313558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.319474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.328989] Lustre: Skipped 1 previous similar message [ 889.336083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.345641] Lustre: Skipped 1 previous similar message [ 893.666464] Lustre: server umount lustre-OST0000 complete [ 899.553687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.553928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.576306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.596443] Lustre: Skipped 1 previous similar message [ 903.353789] Lustre: server umount lustre-MDT0000 complete [ 904.674044] LustreError: 29977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.692223] LustreError: 29977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 907.031989] LustreError: 29958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310799 with bad export cookie 12448904804360929007 [ 907.039186] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.044284] LustreError: 29958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 907.788185] Lustre: server umount lustre-MDT0001 complete [ 919.457558] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 922.805630] Key type lgssc unregistered [ 923.230204] LNet: 33721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.249694] LNetError: 33721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.281608] LNet: Removed LNI 192.168.206.115@tcp [ 924.530979] Key type .llcrypt unregistered [ 924.535991] Key type ._llcrypt unregistered [ 942.762219] Key type ._llcrypt registered [ 942.765724] Key type .llcrypt registered [ 942.898970] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 943.792350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.930614] alg: No test for adler32 (adler32-zlib) [ 944.969877] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 945.250824] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 946.943270] Key type lgssc registered [ 948.409714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.124446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.150582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.675575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.705384] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 968.171684] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.216419] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 972.227327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.552519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.733398] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 986.007571] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.915428] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 999.227740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.605665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.702882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1005.743946] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1012.492314] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.577541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.619463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.628202] Lustre: Skipped 1 previous similar message [ 1033.699823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.710132] Lustre: Skipped 1 previous similar message [ 1038.820170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.828474] Lustre: Skipped 1 previous similar message [ 1040.351131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.498478] Lustre: server umount lustre-OST0000 complete [ 1044.448591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.459945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.473504] Lustre: Skipped 1 previous similar message [ 1044.480992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.060752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.065960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.271134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.466761] Lustre: server umount lustre-MDT0000 complete [ 1059.305083] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.321069] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1062.017717] LustreError: 34975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310954 with bad export cookie 5026317948302729646 [ 1062.023774] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.033885] LustreError: 34975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1062.363322] Lustre: server umount lustre-MDT0001 complete [ 1072.727676] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1075.574273] Key type lgssc unregistered [ 1075.938287] LNet: 38553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.945249] LNetError: 38553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.956351] LNet: Removed LNI 192.168.206.115@tcp [ 1076.881409] Key type .llcrypt unregistered [ 1076.886315] Key type ._llcrypt unregistered [ 1098.079683] Key type ._llcrypt registered [ 1098.082158] Key type .llcrypt registered [ 1098.230811] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1099.002752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1099.136234] alg: No test for adler32 (adler32-zlib) [ 1100.211565] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1100.442823] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1102.104490] Key type lgssc registered [ 1103.208856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.653468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.697480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.038111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.136448] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1121.256594] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.268726] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1124.420322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.784525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1128.854343] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1135.276631] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1140.178081] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1145.700573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1145.939314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.130494] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1156.211925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1156.912849] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1161.698335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.704170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.710861] Lustre: Skipped 1 previous similar message [ 1161.719787] Lustre: Skipped 2 previous similar messages [ 1163.532389] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1163.862874] Lustre: server umount lustre-MDT0000 complete [ 1166.806189] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311059 with bad export cookie 5847610175758483448 [ 1166.812295] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.813332] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1166.820170] LustreError: 39943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.823293] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.823300] Lustre: Skipped 1 previous similar message [ 1166.825867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1168.638113] LustreError: 39944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.937463] LustreError: 39945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.937916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1171.952907] Lustre: Skipped 1 previous similar message [ 1177.057279] LustreError: 41787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.057972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1177.069626] LustreError: 41787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1177.073631] Lustre: Skipped 1 previous similar message [ 1181.151155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1181.383121] Lustre: server umount lustre-MDT0001 complete [ 1201.558830] Lustre: server umount lustre-OST0000 complete [ 1214.015911] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1216.060615] Key type lgssc unregistered [ 1216.340932] LNet: 43806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.350827] LNetError: 43806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.364681] LNet: Removed LNI 192.168.206.115@tcp [ 1216.993882] Key type .llcrypt unregistered [ 1216.996875] Key type ._llcrypt unregistered [ 1234.858456] Key type ._llcrypt registered [ 1234.861416] Key type .llcrypt registered [ 1234.961426] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1235.659183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.694754] alg: No test for adler32 (adler32-zlib) [ 1236.670708] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1236.856893] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1238.495150] Key type lgssc registered [ 1239.318767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.500313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.515417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.786466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.853970] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1254.459793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.710508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.386362] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1262.177263] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1266.009816] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1271.103394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.348563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.713565] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1280.306429] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1290.808712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1295.842047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.843300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.855067] Lustre: Skipped 1 previous similar message [ 1295.863228] Lustre: Skipped 1 previous similar message [ 1297.172681] Lustre: lustre-OST0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1300.962423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.966152] Lustre: Skipped 1 previous similar message [ 1301.784339] Lustre: server umount lustre-OST0000 complete [ 1321.444827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.445973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1321.454980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.463496] Lustre: Skipped 1 previous similar message [ 1324.325424] Lustre: server umount lustre-MDT0000 complete [ 1326.566292] LustreError: 46442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.582913] LustreError: 46442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1327.218462] LustreError: 45183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311220 with bad export cookie 5799482755033837191 [ 1327.221500] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.231269] LustreError: 45183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1333.591930] Lustre: server umount lustre-MDT0001 complete [ 1340.889781] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1342.551212] Key type lgssc unregistered [ 1342.772871] LNet: 48930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.779282] LNetError: 48930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.795728] LNet: Removed LNI 192.168.206.115@tcp [ 1343.400529] Key type .llcrypt unregistered [ 1343.402920] Key type ._llcrypt unregistered [ 1358.499699] Key type ._llcrypt registered [ 1358.501902] Key type .llcrypt registered [ 1358.569950] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1359.130129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.155263] alg: No test for adler32 (adler32-zlib) [ 1360.083150] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1360.226828] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1361.847394] Key type lgssc registered [ 1362.541969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.283904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.299740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.573552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.432367] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1377.052386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.315990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.051564] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1384.950973] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1388.824816] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1393.471904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.637207] LustreError: 52144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.651585] LustreError: 52144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1393.738394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.127938] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1398.777966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1403.075702] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1408.219575] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1410.677175] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1422.292947] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-29 3h48m33s [ 1423.471645] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1426.189075] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1427.258234] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1428.454139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.455350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.457048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.469856] Lustre: Skipped 2 previous similar messages [ 1430.934419] Lustre: server umount lustre-MDT0000 complete [ 1432.312475] LustreError: 51758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.027449] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1435.623113] LustreError: 50326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.634942] LustreError: 50326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1436.782968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.847662] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.020447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.059684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1438.069952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1439.749416] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1442.007755] Lustre: DEBUG MARKER: Wait for df (36692) ... [ 1442.886725] Lustre: DEBUG MARKER: done [ 1447.392933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.396734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.402846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.405408] Lustre: Skipped 5 previous similar messages [ 1448.418651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.539456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.543704] Lustre: Skipped 3 previous similar messages [ 1459.167169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.270996] Lustre: server umount lustre-OST0000 complete [ 1463.776734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.778585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.798167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.804858] Lustre: Skipped 3 previous similar messages [ 1467.804464] Lustre: server umount lustre-MDT0000 complete [ 1469.922418] LustreError: 50320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.932725] LustreError: 50320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1470.198512] LustreError: 50305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311363 with bad export cookie 7510803460643464864 [ 1470.200527] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.204982] LustreError: 50305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1470.389483] Lustre: server umount lustre-MDT0001 complete [ 1476.166827] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1477.638775] Key type lgssc unregistered [ 1477.832708] LNet: 55240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.837557] LNetError: 55240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.846889] LNet: Removed LNI 192.168.206.115@tcp [ 1478.372322] Key type .llcrypt unregistered [ 1478.377033] Key type ._llcrypt unregistered [ 1501.496407] Key type ._llcrypt registered [ 1501.498348] Key type .llcrypt registered [ 1501.585631] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1502.272032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.296865] alg: No test for adler32 (adler32-zlib) [ 1503.247701] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1503.414981] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1505.031206] Key type lgssc registered [ 1505.728777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.975430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.985747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.139358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1512.173735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1512.285751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.115277] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1520.871774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.943699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1520.964805] Lustre: 57427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1521.120624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1524.011519] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1528.685940] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1532.139457] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1536.231635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.371198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1536.497403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.990559] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1544.749870] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1546.290410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1555.217761] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-29 3h50m46s [ 1556.204625] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1560.278167] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1561.667235] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1561.693292] LustreError: 60109:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1581.858760] Lustre: lustre-MDT0000: Client 1b56b054-50fe-469c-9028-94a376a13592 (at 192.168.206.15@tcp) reconnecting [ 1601.743606] LustreError: 60109:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1602.979591] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1604.340663] Lustre: DEBUG MARKER: done [ 1607.651545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1607.651921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.656977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1607.657787] LustreError: Skipped 1 previous similar message [ 1607.671536] Lustre: Skipped 1 previous similar message [ 1612.770224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1612.773235] Lustre: Skipped 2 previous similar messages [ 1613.553108] Lustre: server umount lustre-OST0000 complete [ 1617.888994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.890931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1617.901729] Lustre: Skipped 1 previous similar message [ 1617.910265] Lustre: Skipped 1 previous similar message [ 1624.035443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.039664] Lustre: Skipped 3 previous similar messages [ 1630.175475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1630.178316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.184117] Lustre: Skipped 1 previous similar message [ 1630.296171] Lustre: server umount lustre-MDT0000 complete [ 1632.659163] LustreError: 57418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311525 with bad export cookie 16037143215353396482 [ 1632.664436] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.667289] LustreError: 57418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1632.869388] Lustre: server umount lustre-MDT0001 complete [ 1638.790804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1640.409993] Key type lgssc unregistered [ 1640.604909] LNet: 61753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.612225] LNetError: 61753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.622618] LNet: Removed LNI 192.168.206.115@tcp [ 1641.148476] Key type .llcrypt unregistered [ 1641.150279] Key type ._llcrypt unregistered [ 1666.575519] Key type ._llcrypt registered [ 1666.577448] Key type .llcrypt registered [ 1666.642488] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1667.210400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.244496] alg: No test for adler32 (adler32-zlib) [ 1668.176161] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1668.299268] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1669.927774] Key type lgssc registered [ 1670.631797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1672.810070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1672.836458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.003868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1674.016412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1674.116787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.671509] Lustre: Failing over lustre-MDT0000 [ 1674.826358] Lustre: server umount lustre-MDT0000 complete [ 1678.780328] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:52:50 (1785311570) [ 1682.803706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.019805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.463856] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1690.243490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.302996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1690.322699] Lustre: 64526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.453183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.687251] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1697.053791] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1700.388307] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1703.969651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.075948] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1704.167985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.472590] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1710.131118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1711.527405] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1716.281974] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1720.290651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.295312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.302348] Lustre: Skipped 1 previous similar message [ 1722.052098] Lustre: server umount lustre-MDT0000 complete [ 1724.118420] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311617 with bad export cookie 17939633941781813753 [ 1724.123814] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1724.127731] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1724.307432] Lustre: server umount lustre-MDT0001 complete [ 1726.359711] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1729.001593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1733.717544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.565968] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf8f66205947222d5 [ 1734.581411] Lustre: MGC192.168.206.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1734.781174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.842098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1737.003465] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1740.258730] LustreError: 66377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.260152] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.266893] LustreError: 66377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1740.273496] Lustre: Skipped 1 previous similar message [ 1741.282290] LustreError: 66376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.286103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1741.288958] LustreError: 66376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1741.521212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.624329] Lustre: 63451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311618/real 1785311618] req@000000009bc537f9 x1872034855420544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1743.895348] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1745.747438] Lustre: 63452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311622/real 1785311622] req@0000000005a102e2 x1872034855420672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1746.783894] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311623/real 1785311623] req@00000000177908dd x1872034855420800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1746.920232] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1747.693784] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.574815] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1751.007103] Lustre: 63454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311627/real 1785311627] req@000000008670eaea x1872034855421056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1757.154990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.155201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1757.163084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.172798] Lustre: Skipped 4 previous similar messages [ 1761.917406] Lustre: server umount lustre-MDT0000 complete [ 1762.272518] LustreError: 68294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.285830] LustreError: 68294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1764.155706] LustreError: 64519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311657 with bad export cookie 17939633941781816021 [ 1764.159101] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.166369] LustreError: 64519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1764.326274] Lustre: server umount lustre-MDT0001 complete [ 1767.923706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1773.182447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.303275] LustreError: 70955:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1774.314314] LustreError: 70955:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000acf7e9d6 x1872034855441664/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785311667 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1774.332691] LustreError: 70955:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1774.731908] LustreError: 66377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.781711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.786335] Lustre: Skipped 1 previous similar message [ 1774.826738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1777.281617] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1780.192477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.192524] LustreError: 66376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.205953] Lustre: Skipped 2 previous similar messages [ 1780.226857] LustreError: 66376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1782.246741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.519236] Lustre: 63452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311660/real 1785311660] req@000000004c6f51ad x1872034855441920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1784.865773] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1787.618928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1789.057395] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1792.541159] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1798.118896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1798.123405] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.124564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.138443] Lustre: Skipped 1 previous similar message [ 1802.454035] Lustre: server umount lustre-MDT0000 complete [ 1803.233050] LustreError: 71699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.240492] LustreError: 71699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1804.616380] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311697 with bad export cookie 17939633941781817400 [ 1804.621123] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.622702] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1804.803550] Lustre: server umount lustre-MDT0001 complete [ 1808.420776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1810.406646] Lustre: DEBUG MARKER: files compared the same [ 1821.965283] Lustre: server umount lustre-OST0000 complete [ 1831.031137] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1832.577654] Key type lgssc unregistered [ 1832.760839] LNet: 74244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.764985] LNetError: 74244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.774473] LNet: Removed LNI 192.168.206.115@tcp [ 1833.225193] Key type .llcrypt unregistered [ 1833.226613] Key type ._llcrypt unregistered [ 1845.893367] Key type ._llcrypt registered [ 1845.895069] Key type .llcrypt registered [ 1845.947501] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1846.343637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.371720] alg: No test for adler32 (adler32-zlib) [ 1847.265548] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1847.378065] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1848.975154] Key type lgssc registered [ 1849.453992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.220806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.233803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.525824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.716348] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 1860.917084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.078938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.076784] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 1867.107303] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1869.971825] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1873.431359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.588795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.363664] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 1879.718126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1880.088679] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1890.273964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.280554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.293486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.298463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.299751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.300887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.313814] Lustre: Skipped 1 previous similar message [ 1895.282575] Lustre: server umount lustre-OST0000 complete [ 1901.536394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.536781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.541973] Lustre: Skipped 1 previous similar message [ 1903.271440] Lustre: server umount lustre-MDT0000 complete [ 1905.052136] LustreError: 78217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311797 with bad export cookie 8230640724461671334 [ 1905.055143] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.057738] LustreError: 78217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1905.203540] Lustre: server umount lustre-MDT0001 complete [ 1910.233600] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1911.620344] Key type lgssc unregistered [ 1911.787746] LNet: 79173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.792754] LNetError: 79173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.802077] LNet: Removed LNI 192.168.206.115@tcp [ 1912.207425] Key type .llcrypt unregistered [ 1912.209308] Key type ._llcrypt unregistered [ 1924.445779] Key type ._llcrypt registered [ 1924.447231] Key type .llcrypt registered [ 1924.495634] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1924.998184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.006125] alg: No test for adler32 (adler32-zlib) [ 1925.940429] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1926.091639] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1927.695160] Key type lgssc registered [ 1928.211778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.953081] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.965936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.423484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1952.543258] LustreError: 80541:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 1979.167216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.028947] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1985.745989] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1989.939530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.012208] Lustre: *** cfs_fail_loc=706, val=0*** [ 1990.246725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.290339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1992.279463] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1996.672833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.863986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.785061] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2002.132289] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.575980] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2005.472097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2005.475721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.481623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2007.015848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2007.016184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2007.030165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.550113] Lustre: server umount lustre-OST0000 complete [ 2017.249758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.249933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.251029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.251034] Lustre: Skipped 1 previous similar message [ 2017.265396] Lustre: Skipped 1 previous similar message [ 2019.441659] Lustre: server umount lustre-MDT0000 complete [ 2021.176825] LustreError: 80549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311914 with bad export cookie 16932187862116594251 [ 2021.179704] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.184235] LustreError: 80549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2021.325343] Lustre: server umount lustre-MDT0001 complete [ 2025.884818] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2027.134943] Key type lgssc unregistered [ 2027.280435] LNet: 84150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.284177] LNetError: 84150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.294811] LNet: Removed LNI 192.168.206.115@tcp [ 2027.663565] Key type .llcrypt unregistered [ 2027.665190] Key type ._llcrypt unregistered [ 2038.281101] Key type ._llcrypt registered [ 2038.282362] Key type .llcrypt registered [ 2038.328965] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2038.665910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.704072] alg: No test for adler32 (adler32-zlib) [ 2039.575876] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2039.675469] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2041.263116] Key type lgssc registered [ 2041.680698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2044.733162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2044.741971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.381891] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2050.806798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.064920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.785883] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2056.160052] LustreError: 86229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.166496] LustreError: 86229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2056.297367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.452058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.902272] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2060.577044] LustreError: 86233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.587762] LustreError: 86233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2062.602843] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.608805] Lustre: MGS: Client f6e0fbf8-9b22-49eb-b043-6bed8ba5fc99 (at 0@lo) reconnecting [ 2062.743098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.774172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2064.422718] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2078.177122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.180485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.183257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.187044] Lustre: Skipped 1 previous similar message [ 2080.548707] Lustre: server umount lustre-OST0000 complete [ 2083.297445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.298138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.299107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.299112] Lustre: Skipped 1 previous similar message [ 2083.303470] Lustre: Skipped 1 previous similar message [ 2088.329900] Lustre: server umount lustre-MDT0000 complete [ 2088.417838] LustreError: 86229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.426499] LustreError: 86229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2090.272393] Lustre: server umount lustre-MDT0001 complete [ 2091.960791] Lustre: server umount MGS complete [ 2096.857285] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:59:48 (1785311988) [ 2108.753234] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2112.412773] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2125.541076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.027361] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2131.026246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.121673] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2131.142375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2131.182000] Lustre: lustre-MDT0001: new disk, initializing [ 2131.207838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.218359] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2131.221054] Lustre: 93398:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2131.224436] LustreError: 93398:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2132.660289] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2137.125601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.226165] Lustre: lustre-OST0000: new disk, initializing [ 2137.228449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2137.232176] Lustre: 94165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2137.256184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.255550] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2142.175909] Lustre: 94651:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ca098251 x1872035244444032/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785312051 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2143.102972] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2145.402608] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.407402] Lustre: MGS: Client 75c9fd84-f2b8-45aa-9193-3356c0dfefed (at 0@lo) reconnecting [ 2145.422952] Lustre: 95209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.431609] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2145.436189] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2145.441300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2145.479496] Lustre: lustre-MDT0000: new disk, initializing [ 2145.528419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.540025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2146.591801] Lustre: 95225:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009eaccb6e x1872035244449536/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785312055 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2146.592419] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2146.968349] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2148.642701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2148.646240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2148.649433] Lustre: Skipped 1 previous similar message [ 2148.654134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2166.240637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2166.244392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.250489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2166.252653] Lustre: Skipped 1 previous similar message [ 2168.292585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.295378] Lustre: Skipped 2 previous similar messages [ 2173.410685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2173.413931] Lustre: Skipped 1 previous similar message [ 2180.063194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2180.121458] Lustre: server umount lustre-OST0000 complete [ 2183.648997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.649394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2183.649781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.649784] Lustre: Skipped 2 previous similar messages [ 2183.653373] Lustre: Skipped 2 previous similar messages [ 2183.656126] LustreError: Skipped 1 previous similar message [ 2187.597631] Lustre: server umount lustre-MDT0000 complete [ 2188.768839] LustreError: 93370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.776197] LustreError: 93370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2195.336689] Lustre: server umount lustre-MDT0001 complete [ 2196.759557] Lustre: server umount MGS complete [ 2200.934773] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:01:32 (1785312092) [ 2207.434472] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2208.568786] Key type lgssc unregistered [ 2208.720540] LNet: 97743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.724327] LNetError: 97743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.742375] LNet: Removed LNI 192.168.206.115@tcp [ 2209.090627] Key type .llcrypt unregistered [ 2209.092064] Key type ._llcrypt unregistered [ 2216.694829] Key type ._llcrypt registered [ 2216.696241] Key type .llcrypt registered [ 2216.746402] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2217.260661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.298379] alg: No test for adler32 (adler32-zlib) [ 2218.174818] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2218.278706] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2219.871168] Key type lgssc registered [ 2220.281334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2223.098153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2223.107994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.298191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.762366] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2228.812101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.943572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.353558] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2233.107560] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.174153] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.712091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.850684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.159952] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2243.032250] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2250.150326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2259.424611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.428633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.434265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.449334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.450229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.450654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.459687] Lustre: Skipped 1 previous similar message [ 2262.357546] Lustre: server umount lustre-OST0000 complete [ 2265.056164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.058812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.064166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.847332] Lustre: server umount lustre-MDT0000 complete [ 2270.688651] LustreError: 98940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.695904] LustreError: 98940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2271.263450] LustreError: 100779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312164 with bad export cookie 3552216605316341530 [ 2271.267019] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.267632] LustreError: 100779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2271.383848] Lustre: server umount lustre-MDT0001 complete [ 2275.382639] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2276.512248] Key type lgssc unregistered [ 2276.641211] LNet: 102488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.645092] LNetError: 102488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.655441] LNet: Removed LNI 192.168.206.115@tcp [ 2277.016038] Key type .llcrypt unregistered [ 2277.017569] Key type ._llcrypt unregistered [ 2283.959068] Key type ._llcrypt registered [ 2283.960605] Key type .llcrypt registered [ 2283.997504] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2284.383041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.443324] alg: No test for adler32 (adler32-zlib) [ 2285.302755] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2285.402823] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2286.991125] Key type lgssc registered [ 2287.354390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.478633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.479253] LustreError: 103555:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2289.480740] Lustre: Skipped 1 previous similar message [ 2289.486446] LustreError: 103554:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2289.488683] LustreError: 103555:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2289.492134] LustreError: 103555:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2289.495093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.496316] LustreError: 103555:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2290.644631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.328634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.427411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.922321] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2296.789335] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lsmod [ 2297.890067] LustreError: 104760:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2297.893409] LustreError: 104761:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2297.898220] LustreError: 104760:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2297.902190] LustreError: 104760:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2297.907633] LustreError: 104760:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2297.909591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.091854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.265844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.268708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.271573] Lustre: Skipped 2 previous similar messages [ 2303.275208] Lustre: Skipped 1 previous similar message [ 2305.302807] Lustre: server umount lustre-MDT0000 complete [ 2306.664063] LustreError: 103617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312199 with bad export cookie 475566641398280604 [ 2306.667240] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.668064] LustreError: 103617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2307.359263] LustreError: 104023:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000048300ba5 x1872035502374272/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2307.412087] Lustre: server umount lustre-MDT0001 complete [ 2318.940461] Lustre: server umount lustre-OST0000 complete [ 2321.879116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.035125] LustreError: 105823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.041786] LustreError: 105823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2322.060936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.503618] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2326.529209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.662942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.076810] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2330.746684] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2332.739077] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2335.110082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.223868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.297951] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2339.893819] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2346.473419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2356.707263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.707778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.710655] LustreError: Skipped 1 previous similar message [ 2356.711646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.715897] Lustre: Skipped 1 previous similar message [ 2356.718099] Lustre: Skipped 1 previous similar message [ 2357.847198] Lustre: server umount lustre-OST0000 complete [ 2361.825542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.826054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.830543] Lustre: Skipped 1 previous similar message [ 2361.832552] Lustre: Skipped 1 previous similar message [ 2365.405258] Lustre: server umount lustre-MDT0000 complete [ 2366.815355] LustreError: 105805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312259 with bad export cookie 475566641398281290 [ 2366.818319] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.820044] LustreError: 105805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2366.953645] Lustre: server umount lustre-MDT0001 complete [ 2371.124225] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2372.253199] Key type lgssc unregistered [ 2372.392546] LNet: 109371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.396629] LNetError: 109371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.404407] LNet: Removed LNI 192.168.206.115@tcp [ 2372.734541] Key type .llcrypt unregistered [ 2372.735928] Key type ._llcrypt unregistered [ 2382.372516] Key type ._llcrypt registered [ 2382.374026] Key type .llcrypt registered [ 2382.420538] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2382.878779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2382.907586] alg: No test for adler32 (adler32-zlib) [ 2383.775494] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2383.877492] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2385.463133] Key type lgssc registered [ 2385.876828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.789677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.797395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.959081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.441591] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2394.459224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.601039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.108058] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2398.764636] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2400.806239] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.173484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.322575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.351472] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2407.973370] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.567474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2420.466479] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2421.314811] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2424.801079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.802700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.804703] LustreError: Skipped 1 previous similar message [ 2424.809841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.813402] Lustre: Skipped 1 previous similar message [ 2428.428457] Lustre: server umount lustre-OST0000 complete [ 2429.920302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.920768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.925659] Lustre: Skipped 2 previous similar messages [ 2429.930515] Lustre: Skipped 1 previous similar message [ 2435.040440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.042850] Lustre: Skipped 1 previous similar message [ 2435.864874] Lustre: server umount lustre-MDT0000 complete [ 2437.189841] LustreError: 113316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312330 with bad export cookie 9885072880203220238 [ 2437.193827] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.194828] LustreError: 113316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2437.305377] Lustre: server umount lustre-MDT0001 complete [ 2441.183969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2442.332435] Key type lgssc unregistered [ 2442.469347] LNet: 114420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.471940] LNetError: 114420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.479279] LNet: Removed LNI 192.168.206.115@tcp [ 2442.810344] Key type .llcrypt unregistered [ 2442.812088] Key type ._llcrypt unregistered [ 2450.156081] Key type ._llcrypt registered [ 2450.157572] Key type .llcrypt registered [ 2450.194728] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2450.579804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2450.613704] alg: No test for adler32 (adler32-zlib) [ 2451.480388] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2451.590440] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2453.183139] Key type lgssc registered [ 2453.574936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2456.422841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2456.431296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.585896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.082903] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2462.364405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.508935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.979563] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2466.554582] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.544173] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2470.892039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.011731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.059267] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2475.622602] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.278398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2492.898658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.899246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2492.899386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.903609] Lustre: Skipped 1 previous similar message [ 2494.922493] Lustre: server umount lustre-OST0000 complete [ 2498.017301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.018367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.021705] Lustre: Skipped 1 previous similar message [ 2498.023948] Lustre: Skipped 1 previous similar message [ 2502.364196] Lustre: server umount lustre-MDT0000 complete [ 2503.136601] LustreError: 115618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.144861] LustreError: 115618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2503.738205] LustreError: 115600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312396 with bad export cookie 16149067104935821711 [ 2503.741835] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.742669] LustreError: 115600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2503.865238] Lustre: server umount lustre-MDT0001 complete [ 2507.753787] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2508.888776] Key type lgssc unregistered [ 2509.021671] LNet: 119171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.024437] LNetError: 119171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.035409] LNet: Removed LNI 192.168.206.115@tcp [ 2509.370979] Key type .llcrypt unregistered [ 2509.372399] Key type ._llcrypt unregistered [ 2518.950157] Key type ._llcrypt registered [ 2518.952043] Key type .llcrypt registered [ 2518.992543] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2519.396448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2519.413629] alg: No test for adler32 (adler32-zlib) [ 2520.277798] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2520.385419] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2521.975144] Key type lgssc registered [ 2522.387431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2525.227039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.236460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.417856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.951704] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2531.143867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.299204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.901155] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2535.760907] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2537.898550] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2540.470267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.609845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2542.476054] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2544.999386] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.912372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2558.147693] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2559.507101] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2559.509579] Lustre: Skipped 2 previous similar messages [ 2560.989410] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2560.993230] Lustre: Skipped 2 previous similar messages [ 2566.662055] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2566.675196] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2568.815556] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2568.819168] Lustre: Skipped 2 previous similar messages [ 2575.865577] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp [ 2578.164803] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.15/24@tcp in log lustre-MDT0000 [ 2578.169218] Lustre: Skipped 2 previous similar messages [ 2583.031416] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 2583.032891] Lustre: Skipped 1 previous similar message [ 2588.129177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.131514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.132471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.135527] Lustre: Skipped 1 previous similar message [ 2588.138920] Lustre: Skipped 1 previous similar message [ 2591.950949] Lustre: server umount lustre-OST0000 complete [ 2593.248165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.248549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.252910] Lustre: Skipped 1 previous similar message [ 2593.256666] Lustre: Skipped 1 previous similar message [ 2598.368073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.369701] Lustre: Skipped 1 previous similar message [ 2599.386472] Lustre: server umount lustre-MDT0000 complete [ 2600.724024] LustreError: 120543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312493 with bad export cookie 10141009945137271644 [ 2600.727088] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.728801] LustreError: 120543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.843139] Lustre: server umount lustre-MDT0001 complete [ 2604.471953] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2605.497324] Key type lgssc unregistered [ 2605.612421] LNet: 125802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.614989] LNetError: 125802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.623312] LNet: Removed LNI 192.168.206.115@tcp [ 2605.927624] Key type .llcrypt unregistered [ 2605.929138] Key type ._llcrypt unregistered [ 2616.165167] Key type ._llcrypt registered [ 2616.166795] Key type .llcrypt registered [ 2616.205862] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2616.587955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.620880] alg: No test for adler32 (adler32-zlib) [ 2617.491750] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2617.588733] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2619.167208] Key type lgssc registered [ 2619.539621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.285357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.293878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.438915] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2623.440563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.447153] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.15/24@tcp [ 2624.885339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2627.770327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.883833] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2627.892907] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 2627.909249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2629.346945] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2631.846729] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2633.746971] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2636.189045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.324808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.339111] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2640.786572] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2649.640379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2654.689072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.692425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.692951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2654.695095] Lustre: Skipped 1 previous similar message [ 2656.533193] Lustre: server umount lustre-OST0000 complete [ 2658.783979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2658.788187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.795108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.797602] Lustre: Skipped 1 previous similar message [ 2659.808584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.809315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.989675] Lustre: server umount lustre-MDT0000 complete [ 2661.245227] LustreError: 127319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312554 with bad export cookie 7049458111164960308 [ 2661.248075] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.249798] LustreError: 127319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2661.373695] Lustre: server umount lustre-MDT0001 complete [ 2664.972873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2665.971546] Key type lgssc unregistered [ 2666.107638] LNet: 130942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.109937] LNetError: 130942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.118296] LNet: Removed LNI 192.168.206.115@tcp [ 2666.441477] Key type .llcrypt unregistered [ 2666.442809] Key type ._llcrypt unregistered [ 2675.200844] Key type ._llcrypt registered [ 2675.201793] Key type .llcrypt registered [ 2675.241292] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2681.461714] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2681.795687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.823183] alg: No test for adler32 (adler32-zlib) [ 2682.672727] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2682.753529] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2684.335142] Key type lgssc registered [ 2684.668572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.906254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2695.912946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.993963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2697.004619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2697.039187] Lustre: lustre-MDT0000: new disk, initializing [ 2697.062044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.068646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2698.446217] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2702.583913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.631844] Lustre: 134313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2702.656846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2702.658890] Lustre: Skipped 1 previous similar message [ 2702.694314] Lustre: lustre-MDT0001: new disk, initializing [ 2702.728335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.742835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2702.746810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2704.041977] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2707.754576] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.520681] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2711.707201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.810531] Lustre: lustre-OST0000: new disk, initializing [ 2711.812309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2711.814615] Lustre: 136280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2711.843200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.632641] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2714.036509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2714.039341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2714.068117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2717.218665] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.598098] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2718.667235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2719.937672] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2720.014098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2724.320580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.322752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.322899] LustreError: Skipped 1 previous similar message [ 2724.327078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.330364] Lustre: Skipped 1 previous similar message [ 2726.735689] Lustre: server umount lustre-OST0000 complete [ 2728.416046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.418679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.422566] Lustre: Skipped 1 previous similar message [ 2728.424812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.440417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.441431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.560149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.563639] Lustre: Skipped 2 previous similar messages [ 2739.680155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.682746] Lustre: Skipped 1 previous similar message [ 2742.239106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2742.301367] Lustre: server umount lustre-MDT0000 complete [ 2743.628457] LustreError: 134304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312636 with bad export cookie 2301013905569774573 [ 2743.632151] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.633597] LustreError: 134304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.753924] Lustre: server umount lustre-MDT0001 complete [ 2746.672466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.854824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.325733] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2751.302426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.417361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.791986] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2755.355340] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2757.255336] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2759.685392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.805252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.767819] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2764.308735] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2765.783747] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2771.977130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2773.320749] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2773.388702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2776.576969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.636021] Lustre: lustre-OST0001: new disk, initializing [ 2776.638129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2776.642064] Lustre: 141909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2776.668250] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.456902] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2781.604780] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2784.775791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2784.779733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2784.808986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2785.744440] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2787.035959] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2787.108443] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2788.744176] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9233d2343000.ost_server_uuid 50 [ 2789.203496] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9233d2343000.ost_server_uuid in FULL state after 0 sec [ 2794.978904] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.980993] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.986523] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2794.988294] Lustre: Skipped 1 previous similar message [ 2797.052407] Lustre: server umount lustre-OST0001 complete [ 2800.099859] LustreError: 140318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.100676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.104358] LustreError: 140318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2800.107409] Lustre: Skipped 2 previous similar messages [ 2804.303142] Lustre: server umount lustre-OST0000 complete [ 2807.775774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.778966] LustreError: Skipped 1 previous similar message [ 2810.336840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.341919] Lustre: Skipped 1 previous similar message [ 2811.736364] Lustre: server umount lustre-MDT0000 complete [ 2813.072291] LustreError: 138477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312705 with bad export cookie 2301013905569775434 [ 2813.074753] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2813.079581] LustreError: 138477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2813.202462] Lustre: server umount lustre-MDT0001 complete [ 2819.616553] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2820.587670] Key type lgssc unregistered [ 2820.708396] LNet: 144935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.710671] LNetError: 144935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2820.717272] LNet: Removed LNI 192.168.206.115@tcp [ 2821.014620] Key type .llcrypt unregistered [ 2821.015854] Key type ._llcrypt unregistered [ 2833.482799] Key type ._llcrypt registered [ 2833.483793] Key type .llcrypt registered [ 2833.520692] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2839.485118] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2839.794662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.839439] alg: No test for adler32 (adler32-zlib) [ 2840.683572] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2840.766701] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2842.343093] Key type lgssc registered [ 2842.663360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2853.380993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2853.387410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.471270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2854.480960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2854.519048] Lustre: lustre-MDT0000: new disk, initializing [ 2854.538389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.544677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2855.840354] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2859.851941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.893943] Lustre: 149060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2859.915889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2859.917920] Lustre: Skipped 1 previous similar message [ 2859.953928] Lustre: lustre-MDT0001: new disk, initializing [ 2859.985206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.995813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2859.999102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2861.231448] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2864.826818] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2866.526282] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2868.675596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.772740] Lustre: lustre-OST0000: new disk, initializing [ 2868.774269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2868.777852] Lustre: 151026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.805114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.542804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2873.854256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2873.858488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2873.894555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2874.037133] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2875.965540] Lustre: Failing over lustre-OST0000 [ 2876.031746] Lustre: server umount lustre-OST0000 complete [ 2888.879967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.987304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.990734] Lustre: Skipped 1 previous similar message [ 2888.996584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.002433] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2890.494762] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2890.768197] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2891.952931] Lustre: Failing over lustre-MDT0000 [ 2892.044707] Lustre: server umount lustre-MDT0000 complete [ 2894.304444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.308545] Lustre: Skipped 2 previous similar messages [ 2895.327073] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785312771/real 1785312771] req@00000000470eae2d x1872036084343552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785312787 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2899.423122] Lustre: 147288:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785312776/real 1785312776] req@00000000b185cbac x1872036084345344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785312792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2899.424325] LustreError: 149870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.433592] LustreError: 149870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2900.728223] LustreError: 149064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.544219] LustreError: 149070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.549734] LustreError: 149070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2904.876290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.912309] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.916869] Lustre: 149062:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2904.977585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2904.993866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.021342] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2905.848540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2906.311728] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2910.180354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2910.185981] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2910.206982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2915.297965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.299612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.300180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.301882] LustreError: Skipped 1 previous similar message [ 2915.305509] Lustre: Skipped 1 previous similar message [ 2917.261746] Lustre: server umount lustre-OST0000 complete [ 2920.416501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2920.416864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.417637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.417643] Lustre: Skipped 1 previous similar message [ 2920.428041] Lustre: Skipped 1 previous similar message [ 2924.559322] Lustre: server umount lustre-MDT0000 complete [ 2925.535859] LustreError: 149070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.539729] LustreError: 149070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2925.766166] LustreError: 149854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312818 with bad export cookie 8056728498883828277 [ 2925.769647] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.770654] LustreError: 149854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2925.883989] Lustre: server umount lustre-MDT0001 complete [ 2929.513340] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2930.489987] Key type lgssc unregistered [ 2930.621388] LNet: 154670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2930.624366] LNetError: 154670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2930.636285] LNet: Removed LNI 192.168.206.115@tcp [ 2930.964838] Key type .llcrypt unregistered [ 2930.966402] Key type ._llcrypt unregistered [ 2950.853137] Key type ._llcrypt registered [ 2950.854110] Key type .llcrypt registered [ 2950.893767] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2951.430390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.460051] alg: No test for adler32 (adler32-zlib) [ 2952.315271] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2952.413423] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2953.991160] Key type lgssc registered [ 2954.398865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.999565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2957.007892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.150129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.379342] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2962.978932] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2965.043381] LustreError: 156215:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2965.047199] LustreError: 156215:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2965.049966] LustreError: 156215:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2965.176828] Lustre: server umount lustre-MDT0000 complete [ 2970.044856] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2973.170201] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2984.210569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.303746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2984.315480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2984.356497] Lustre: lustre-MDT0000: new disk, initializing [ 2984.380363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.388438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2985.749730] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2989.944911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.981648] Lustre: 159510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2990.001655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2990.003400] Lustre: Skipped 1 previous similar message [ 2990.038232] Lustre: lustre-MDT0001: new disk, initializing [ 2990.066714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.079105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2990.082362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2991.382745] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2995.006049] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2996.886309] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2999.046358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.142324] Lustre: lustre-OST0000: new disk, initializing [ 2999.144421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2999.148124] Lustre: 161474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2999.180533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.247831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3000.250842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3000.285118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3001.032127] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3004.752649] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.565319] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3007.130601] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:14:59 (1785312899) [ 3077.087635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.087692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.087910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3077.096214] Lustre: Skipped 1 previous similar message [ 3079.391869] Lustre: server umount lustre-MDT0000 complete [ 3080.634620] LustreError: 159503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312973 with bad export cookie 10518392773567508648 [ 3080.635496] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.638879] LustreError: 159503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3082.207534] LustreError: 159517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.207560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.215071] LustreError: 159517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3082.221085] Lustre: Skipped 1 previous similar message [ 3082.225565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3082.227631] Lustre: Skipped 2 previous similar messages [ 3085.215441] LustreError: 162195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.215454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3086.821091] Lustre: server umount lustre-MDT0001 complete [ 3094.219441] Lustre: server umount lustre-OST0000 complete [ 3097.373688] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3100.645674] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3112.798875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.872574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3112.882569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3112.917419] Lustre: lustre-MDT0000: new disk, initializing [ 3112.934280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.939589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3114.177134] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3118.042752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.079279] Lustre: 166102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3118.097117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3118.098422] Lustre: Skipped 1 previous similar message [ 3118.129540] Lustre: lustre-MDT0001: new disk, initializing [ 3118.167322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3118.170680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3119.400402] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3122.902902] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3124.566744] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3126.582868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.680204] Lustre: lustre-OST0000: new disk, initializing [ 3126.681686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3126.683715] Lustre: 168072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3126.706790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.709265] Lustre: Skipped 1 previous similar message [ 3128.444762] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3132.175224] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3133.612281] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3136.504111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3136.507897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3136.539395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3136.745239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3138.088174] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3138.155925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3141.601738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.602469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.605293] LustreError: Skipped 1 previous similar message [ 3141.608824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.612024] Lustre: Skipped 1 previous similar message [ 3144.847683] Lustre: server umount lustre-OST0000 complete [ 3146.720219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.720729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3146.724962] Lustre: Skipped 2 previous similar messages [ 3152.218541] Lustre: server umount lustre-MDT0000 complete [ 3153.498983] LustreError: 166139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313046 with bad export cookie 10518392773567667464 [ 3153.502197] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3153.505640] LustreError: 166139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3153.628786] Lustre: server umount lustre-MDT0001 complete [ 3156.456319] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:17:28 (1785313048) [ 3161.799313] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3164.747122] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3175.395752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.466607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3175.475089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3175.520472] Lustre: lustre-MDT0000: new disk, initializing [ 3175.540108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.545976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3176.792156] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3180.926467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.970372] Lustre: 172843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3181.071889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3182.396208] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3186.087971] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3187.946573] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3190.172678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.297803] Lustre: 174817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3192.357241] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3195.988704] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3197.950697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3197.991872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3203.041648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.041811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.046095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.047600] LustreError: Skipped 1 previous similar message [ 3203.050646] Lustre: Skipped 4 previous similar messages [ 3217.375122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.429447] Lustre: server umount lustre-OST0000 complete [ 3221.984032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3221.987851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.992149] Lustre: Skipped 1 previous similar message [ 3221.994098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3221.996216] Lustre: Skipped 4 previous similar messages [ 3233.247206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3233.317890] Lustre: server umount lustre-MDT0000 complete [ 3233.760498] LustreError: 172853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.769132] LustreError: 172853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3234.676097] LustreError: 174816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313127 with bad export cookie 10518392773567668318 [ 3234.679142] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.680036] LustreError: 174816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3234.804243] Lustre: server umount lustre-MDT0001 complete [ 3237.844198] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:18:49 (1785313129) [ 3243.048917] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3246.027733] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3257.540053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.637732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3257.649302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3257.651902] Lustre: Skipped 3 previous similar messages [ 3257.695412] Lustre: lustre-MDT0000: new disk, initializing [ 3257.697643] Lustre: Skipped 2 previous similar messages [ 3257.719913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.721993] Lustre: Skipped 2 previous similar messages [ 3257.726582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3257.728910] Lustre: Skipped 2 previous similar messages [ 3259.127515] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3263.450690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.498440] Lustre: 179132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3263.601295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3264.981281] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3268.749523] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3270.597948] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3272.880746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.003334] Lustre: 181109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3274.948421] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3278.560991] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.451459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3287.521491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3287.522208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.523569] LustreError: Skipped 1 previous similar message [ 3287.526475] Lustre: Skipped 1 previous similar message [ 3287.527022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.529931] Lustre: Skipped 5 previous similar messages [ 3291.919179] Lustre: server umount lustre-OST0000 complete [ 3299.220266] Lustre: server umount lustre-MDT0000 complete [ 3300.378349] LustreError: 179931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313193 with bad export cookie 10518392773567669389 [ 3300.380985] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3300.383166] LustreError: 179931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3300.507029] Lustre: server umount lustre-MDT0001 complete [ 3303.930245] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3304.882644] Key type lgssc unregistered [ 3305.004400] LNet: 182964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.007384] LNetError: 182964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.016273] LNet: Removed LNI 192.168.206.115@tcp [ 3305.307904] Key type .llcrypt unregistered [ 3305.309144] Key type ._llcrypt unregistered [ 3313.763758] Key type ._llcrypt registered [ 3313.765254] Key type .llcrypt registered [ 3313.805895] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3314.294482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3314.306503] alg: No test for adler32 (adler32-zlib) [ 3315.153896] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 3315.234709] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3316.815147] Key type lgssc registered [ 3317.138795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3319.515198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3319.520144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.644470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.862707] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3324.712619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.840455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3326.238599] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3328.664683] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3330.574860] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3332.805026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3332.924056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.888776] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3337.307930] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.258377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3360.740205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.742398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.743525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.744428] Lustre: Skipped 1 previous similar message [ 3360.749829] LustreError: Skipped 1 previous similar message [ 3365.326481] Lustre: server umount lustre-OST0000 complete [ 3370.976769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3370.976866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.977528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.977533] Lustre: Skipped 1 previous similar message [ 3370.987152] Lustre: Skipped 1 previous similar message [ 3372.690828] Lustre: server umount lustre-MDT0000 complete [ 3373.954248] LustreError: 184339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313266 with bad export cookie 9177730004843793531 [ 3373.957299] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.959235] LustreError: 184339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3374.073968] Lustre: server umount lustre-MDT0001 complete [ 3377.480403] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3378.492940] Key type lgssc unregistered [ 3378.609360] LNet: 187909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3378.611872] LNetError: 187909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3378.622366] LNet: Removed LNI 192.168.206.115@tcp [ 3378.893949] Key type .llcrypt unregistered [ 3378.895160] Key type ._llcrypt unregistered [ 3388.013837] Key type ._llcrypt registered [ 3388.014767] Key type .llcrypt registered [ 3388.054909] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3388.346988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3388.371310] alg: No test for adler32 (adler32-zlib) [ 3389.220371] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 3389.299765] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3390.879113] Key type lgssc registered [ 3391.231702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.675858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.682332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.814730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.053481] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3398.608817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.726698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3399.962581] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3402.279535] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3403.999328] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3406.086775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.193518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.963782] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3410.221862] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3419.493447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3423.711766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.714457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.718365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.737989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.738156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.739150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.745855] Lustre: Skipped 1 previous similar message [ 3426.050740] Lustre: lustre-OST0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 3429.201242] Lustre: server umount lustre-OST0000 complete [ 3431.488255] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9233c6e89000.ost_server_uuid 50 [ 3432.003457] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3444.703822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3444.707035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.711117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.152877] Lustre: server umount lustre-MDT0000 complete [ 3449.312327] LustreError: 189321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.320300] LustreError: 189321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3449.328220] LustreError: 189283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313342 with bad export cookie 10072262873796886932 [ 3449.331838] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.332366] LustreError: 189283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3449.438558] Lustre: server umount lustre-MDT0001 complete [ 3452.066834] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:22:24 (1785313344) [ 3454.190168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.338281] LustreError: 193215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.360270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.655634] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3458.317964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.430850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.780154] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3462.000829] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.699302] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3465.771391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.893413] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3470.469204] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3472.575595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.645417] Lustre: lustre-OST0001: new disk, initializing [ 3472.647590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3472.649633] Lustre: 196054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3472.677550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.680205] Lustre: Skipped 1 previous similar message [ 3474.432834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3474.437308] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3474.468437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3474.557633] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3478.139677] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3479.523175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.526840] Lustre: Skipped 1 previous similar message [ 3479.528542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.528670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3479.532813] Lustre: Skipped 2 previous similar messages [ 3485.046083] Lustre: server umount lustre-OST0000 complete [ 3485.666277] LustreError: 195038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.673472] LustreError: 195038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3486.868542] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3486.936707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3488.234078] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3488.303550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3488.504986] LustreError: 196379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.624734] LustreError: 196978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.629629] LustreError: 196978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3499.999965] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.000095] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.001947] LustreError: Skipped 1 previous similar message [ 3500.008302] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3500.010888] Lustre: Skipped 4 previous similar messages [ 3504.332616] Lustre: server umount lustre-OST0001 complete [ 3511.698934] Lustre: server umount lustre-MDT0000 complete [ 3512.796863] LustreError: 193951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313405 with bad export cookie 10072262873796887989 [ 3512.799239] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3512.801320] LustreError: 193951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3512.906079] Lustre: server umount lustre-MDT0001 complete [ 3520.252161] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:23:32 (1785313412) [ 3522.367526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.454384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3522.465845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3522.533088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.783807] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3526.383806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.411912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3526.423222] Lustre: 199466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3527.878397] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3530.139634] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3533.344473] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3535.573000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.639730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3536.526768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3537.604606] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3540.041041] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3542.364576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.387737] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3544.088084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3544.504883] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3547.119750] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3549.157585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.161543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.162723] Lustre: Skipped 4 previous similar messages [ 3549.167870] Lustre: Skipped 7 previous similar messages [ 3554.169350] Lustre: server umount lustre-OST0000 complete [ 3554.276459] LustreError: 201315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.281356] LustreError: 201315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3569.636000] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.637160] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.638611] LustreError: Skipped 1 previous similar message [ 3570.771948] Lustre: server umount lustre-OST0001 complete [ 3578.257429] Lustre: server umount lustre-MDT0000 complete [ 3579.489166] LustreError: 200214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313472 with bad export cookie 10072262873796889151 [ 3579.492931] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.494086] LustreError: 200214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3579.611534] Lustre: server umount lustre-MDT0001 complete [ 3587.024388] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:24:39 (1785313479) [ 3592.004436] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3595.041907] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3606.009603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.083976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3606.094571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3606.131482] Lustre: lustre-MDT0000: new disk, initializing [ 3606.151518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.154335] Lustre: Skipped 3 previous similar messages [ 3606.160351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3607.411446] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3611.375588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.413780] Lustre: 207527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3611.436721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3611.438553] Lustre: Skipped 1 previous similar message [ 3611.475110] Lustre: lustre-MDT0001: new disk, initializing [ 3611.516142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3611.518779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3612.821233] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3616.430483] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3618.196946] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3620.430321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.517075] Lustre: lustre-OST0000: new disk, initializing [ 3620.518600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3620.521291] Lustre: 209491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.251433] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3625.738015] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3627.017937] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3628.532193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3628.535368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3628.564179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3629.120065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3630.389320] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3630.458782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3633.634838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3633.635405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.638114] LustreError: Skipped 2 previous similar messages [ 3633.641393] Lustre: Skipped 3 previous similar messages [ 3633.641994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3633.646553] Lustre: Skipped 7 previous similar messages [ 3637.138713] Lustre: server umount lustre-OST0000 complete [ 3644.570634] Lustre: server umount lustre-MDT0000 complete [ 3645.820150] LustreError: 207561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313538 with bad export cookie 10072262873796890915 [ 3645.822938] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.825081] LustreError: 207561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3645.937591] Lustre: server umount lustre-MDT0001 complete [ 3648.719815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.872407] LustreError: 211716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.878101] LustreError: 211716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3650.320134] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3653.253923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.618815] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3656.803534] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3658.601308] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.989024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.071533] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3665.537081] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.033426] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3674.227106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3675.453969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3675.519293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3678.688612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.691429] LustreError: Skipped 1 previous similar message [ 3682.185920] Lustre: server umount lustre-OST0000 complete [ 3684.114532] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3685.456354] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3708.935104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.064286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.067322] Lustre: Skipped 5 previous similar messages [ 3711.085301] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3713.879536] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3714.489474] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3717.640224] Lustre: server umount lustre-OST0000 complete [ 3720.159796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.163337] Lustre: Skipped 5 previous similar messages [ 3720.165642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.167923] Lustre: Skipped 6 previous similar messages [ 3725.009945] Lustre: server umount lustre-MDT0000 complete [ 3726.197038] LustreError: 211742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313619 with bad export cookie 10072262873796891783 [ 3726.199375] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.203133] LustreError: 211742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3726.312424] Lustre: server umount lustre-MDT0001 complete [ 3729.126195] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:27:01 (1785313621) [ 3731.266430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.404404] LustreError: 217721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.706150] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3735.429514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.769675] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3738.939903] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3740.599820] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3742.700192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3744.627033] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3746.980030] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.418885] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3756.657364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3758.067958] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3758.139244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3760.286210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3760.358103] Lustre: lustre-OST0001: new disk, initializing [ 3760.360328] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3760.364062] Lustre: 221146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3762.335993] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3766.258158] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3767.632803] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3768.317682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3768.321711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3768.344459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3768.714705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3770.000939] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3770.071580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3773.414167] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.417335] LustreError: Skipped 2 previous similar messages [ 3776.762568] Lustre: server umount lustre-OST0001 complete [ 3778.611465] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3778.686312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3780.072601] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3780.146646] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3795.705503] LustreError: 220496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.711152] LustreError: 220496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3803.744449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3805.754243] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3808.121801] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.663237] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3809.254244] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3809.257962] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3809.258037] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3809.265228] Lustre: Skipped 1 previous similar message [ 3815.420912] Lustre: server umount lustre-OST0001 complete [ 3833.549986] Lustre: server umount lustre-OST0000 complete [ 3849.183092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3849.239419] Lustre: server umount lustre-MDT0000 complete [ 3850.444078] LustreError: 217702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313743 with bad export cookie 10072262873796892714 [ 3850.447411] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3850.449166] LustreError: 217702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3856.621286] Lustre: server umount lustre-MDT0001 complete [ 3863.998609] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:29:16 (1785313756) [ 3866.040656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.121830] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3866.132196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3866.198885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.200950] Lustre: Skipped 5 previous similar messages [ 3867.459338] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3870.069897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.104660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3870.112961] Lustre: 226386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3871.473709] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3873.742831] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.508202] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3877.701658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.764244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3879.644283] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3881.954884] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3885.618878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3887.637646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3889.954887] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3890.485407] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3891.869378] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3891.955147] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3893.289837] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3893.353162] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3895.300558] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9233d2340000.ost_server_uuid 50 [ 3895.838159] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9233d2340000.ost_server_uuid in FULL state after 0 sec [ 3896.226075] Lustre: Permanently deactivating lustre-OST0001 [ 3896.232084] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3896.263373] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3897.205342] Lustre: Permanently reactivating lustre-OST0001 [ 3904.377806] Lustre: server umount lustre-OST0001 complete [ 3906.016522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3906.016744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.019051] LustreError: Skipped 5 previous similar messages [ 3906.023269] Lustre: Skipped 9 previous similar messages [ 3906.023862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3906.029510] Lustre: Skipped 18 previous similar messages [ 3911.759610] Lustre: server umount lustre-OST0000 complete [ 3922.427656] Lustre: server umount lustre-MDT0000 complete [ 3923.699218] LustreError: 226376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313816 with bad export cookie 10072262873796893988 [ 3923.705226] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3926.495253] LustreError: 231145:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3926.496041] LustreError: 227133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.500653] LustreError: 226388:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3926.505963] LustreError: 227133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3926.623734] Lustre: server umount lustre-MDT0001 complete [ 3934.230986] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:30:26 (1785313826) [ 3936.575606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.663423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3936.672509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3936.676313] Lustre: Skipped 5 previous similar messages [ 3937.938273] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3940.670485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.701911] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3940.703578] Lustre: Skipped 1 previous similar message [ 3940.711939] Lustre: 233073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3942.051187] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3944.379486] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3946.035282] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3948.108817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3948.183269] Lustre: Permanently deactivating lustre-OST0000 [ 3948.187066] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3950.086920] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3952.321184] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3954.396200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3954.468137] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3954.469847] Lustre: Skipped 1 previous similar message [ 3954.520085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3954.560247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3956.337026] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3958.532705] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3959.106679] Lustre: Permanently reactivating lustre-OST0000 [ 3963.894902] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3963.915712] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3963.918305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3963.921195] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3976.509171] Lustre: server umount lustre-OST0001 complete [ 3983.762224] Lustre: server umount lustre-OST0000 complete [ 3991.257126] Lustre: server umount lustre-MDT0000 complete [ 3992.436437] LustreError: 233815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313885 with bad export cookie 10072262873796895605 [ 3992.440816] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.441982] LustreError: 233815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3992.566900] Lustre: server umount lustre-MDT0001 complete [ 3996.090857] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3997.110862] Key type lgssc unregistered [ 3997.241404] LNet: 237940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3997.244596] LNetError: 237940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3997.250296] LNet: Removed LNI 192.168.206.115@tcp [ 3997.551813] Key type .llcrypt unregistered [ 3997.553456] Key type ._llcrypt unregistered [ 4005.957373] Key type ._llcrypt registered [ 4005.958291] Key type .llcrypt registered [ 4005.988480] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4006.361031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4006.374206] alg: No test for adler32 (adler32-zlib) [ 4007.226491] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4007.310934] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4008.887127] Key type lgssc registered [ 4009.222223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4012.425510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4012.430330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.592427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.889164] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4017.559777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.600982] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4017.610870] Lustre: 239373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4017.614963] Lustre: Permanently deactivating lustre-MDT0001 [ 4017.619966] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4017.727430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.990724] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4019.818533] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4019.820445] LustreError: 239415:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4019.825316] LustreError: 239415:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4019.829327] LustreError: 239415:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4021.359711] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.230341] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4025.420702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4025.521596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.298620] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4027.555124] LustreError: 241224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.557659] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4027.559930] LustreError: 241224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4029.593462] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4031.826734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4031.902705] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4033.845672] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4036.181793] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4037.094466] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4038.123311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4038.123368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4041.917728] Lustre: Permanently reactivating lustre-MDT0001 [ 4041.924556] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4041.927591] Lustre: Skipped 2 previous similar messages [ 4045.824085] Lustre: lustre-MDT0001: Client 15312121-8d2d-4fa4-8aa7-c9a65ff80c15 (at 192.168.206.15@tcp) reconnecting [ 4047.767627] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4048.302906] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4049.964521] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4050.033875] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4051.358669] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4051.432716] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4052.263911] Lustre: Permanently deactivating lustre-MDT0001 [ 4052.270051] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4052.272288] Lustre: Skipped 2 previous similar messages [ 4059.131479] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4063.716196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.716346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4063.719331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4063.721564] Lustre: Skipped 2 previous similar messages [ 4068.114742] Lustre: server umount lustre-MDT0000 complete [ 4068.833246] LustreError: 239401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.838900] LustreError: 239401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4069.318226] LustreError: 239363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313962 with bad export cookie 17096890331031100043 [ 4069.323121] LustreError: 239363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4069.323167] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.434980] Lustre: server umount lustre-MDT0001 complete [ 4076.794127] Lustre: server umount lustre-OST0000 complete [ 4080.653296] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:32:52 (1785313972) [ 4083.860292] Lustre: server umount lustre-OST0001 complete [ 4085.862316] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4089.007063] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4100.333667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.407618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4100.411753] Lustre: Skipped 2 previous similar messages [ 4100.419608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4100.456664] Lustre: lustre-MDT0000: new disk, initializing [ 4100.476347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.482902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4101.754218] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4105.689088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.728232] Lustre: 247673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4105.758760] LustreError: 247684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.761343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4105.766876] Lustre: Skipped 1 previous similar message [ 4105.803271] Lustre: lustre-MDT0001: new disk, initializing [ 4105.841038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4105.843667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4107.122876] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4110.683466] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4112.473334] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4114.637026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4114.740244] Lustre: lustre-OST0000: new disk, initializing [ 4114.742678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4114.746174] Lustre: 249641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4114.773637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.776164] Lustre: Skipped 1 previous similar message [ 4116.596246] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4116.981607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4116.985913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4117.024035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4120.167263] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4120.721444] LustreError: 247679:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4124.874880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4124.945028] Lustre: lustre-OST0001: new disk, initializing [ 4124.947436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4124.951150] Lustre: 250758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4126.906536] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4130.723522] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4130.815139] LustreError: 247679:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4130.825349] LustreError: 247709:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4131.286709] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4132.354352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4132.359594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4132.397110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4140.919061] LustreError: 247709:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4142.563304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.563758] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4142.564510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4142.564516] Lustre: Skipped 3 previous similar messages [ 4142.568026] Lustre: Skipped 2 previous similar messages [ 4142.570243] LustreError: Skipped 1 previous similar message [ 4147.652546] Lustre: server umount lustre-OST0001 complete [ 4147.682664] LustreError: 249626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.687436] LustreError: 249626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4152.805559] LustreError: 250068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.809813] LustreError: 250068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4163.045085] LustreError: 249625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.045302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4163.045317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.047049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4163.047061] Lustre: Skipped 1 previous similar message [ 4163.049024] LustreError: 249625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4163.060543] LustreError: Skipped 1 previous similar message [ 4165.648189] Lustre: server umount lustre-OST0000 complete [ 4167.647739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4167.649583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.652449] Lustre: Skipped 1 previous similar message [ 4167.653922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.655387] Lustre: Skipped 1 previous similar message [ 4173.021578] Lustre: server umount lustre-MDT0000 complete [ 4174.316282] LustreError: 247711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314067 with bad export cookie 17096890331031102857 [ 4174.320759] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.320989] LustreError: 247711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4180.534739] Lustre: server umount lustre-MDT0001 complete [ 4184.368230] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4185.316149] Key type lgssc unregistered [ 4185.453323] LNet: 252786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4185.456820] LNetError: 252786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4185.464310] LNet: Removed LNI 192.168.206.115@tcp [ 4185.791245] Key type .llcrypt unregistered [ 4185.792398] Key type ._llcrypt unregistered [ 4200.154105] Key type ._llcrypt registered [ 4200.155384] Key type .llcrypt registered [ 4200.188906] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4200.515500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4200.551068] alg: No test for adler32 (adler32-zlib) [ 4201.398938] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4201.485990] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4203.063114] Key type lgssc registered [ 4203.386919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4205.848576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4205.854728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.936289] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4206.943721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4206.999688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.262919] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4210.905439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.932309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4210.940826] Lustre: 254969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4211.027906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4212.294948] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4214.693288] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.400984] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4218.592631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4218.659569] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4218.718114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.537147] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4222.896561] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.581883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4267.489894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4267.490223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.492078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4267.492328] LustreError: Skipped 1 previous similar message [ 4267.497204] Lustre: Skipped 1 previous similar message [ 4272.610889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.613223] Lustre: Skipped 2 previous similar messages [ 4273.193466] Lustre: server umount lustre-OST0000 complete [ 4274.682198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4278.857953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.966922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.001858] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4283.548275] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.318779] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.324956] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4297.325044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4318.176862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.178200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.179258] LustreError: Skipped 1 previous similar message [ 4318.183232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.185865] Lustre: Skipped 1 previous similar message [ 4323.096321] Lustre: server umount lustre-OST0000 complete [ 4328.416431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.417075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.420935] Lustre: Skipped 2 previous similar messages [ 4330.456576] Lustre: server umount lustre-MDT0000 complete [ 4331.595118] LustreError: 257557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314224 with bad export cookie 10486599079078593890 [ 4331.598132] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.598788] LustreError: 257557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4331.722430] Lustre: server umount lustre-MDT0001 complete [ 4335.147050] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4336.076541] Key type lgssc unregistered [ 4336.200425] LNet: 260380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4336.203287] LNetError: 260380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4336.211237] LNet: Removed LNI 192.168.206.115@tcp [ 4336.496927] Key type .llcrypt unregistered [ 4336.497967] Key type ._llcrypt unregistered [ 4345.428856] Key type ._llcrypt registered [ 4345.429792] Key type .llcrypt registered [ 4345.465382] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4345.759764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4345.789069] alg: No test for adler32 (adler32-zlib) [ 4346.631782] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4346.712552] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4348.287086] Key type lgssc registered [ 4348.614865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4350.974657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4350.981578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.137933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.591223] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4356.401683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.545857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4357.898782] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4360.380157] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4362.020110] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4364.170516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.268064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.047290] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4368.290974] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4377.573294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4386.783825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4386.787442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.793642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4387.808432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4387.809474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.809581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4392.015557] Lustre: server umount lustre-OST0000 complete [ 4397.535888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4397.538894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.543892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.545642] Lustre: Skipped 1 previous similar message [ 4399.309891] Lustre: server umount lustre-MDT0000 complete [ 4400.455257] LustreError: 261753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314293 with bad export cookie 10182008442664569273 [ 4400.459655] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4400.459790] LustreError: 261753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4400.586689] Lustre: server umount lustre-MDT0001 complete [ 4403.676309] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4404.714405] Key type lgssc unregistered [ 4404.832368] LNet: 266097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4404.835283] LNetError: 266097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4404.847353] LNet: Removed LNI 192.168.206.115@tcp [ 4405.141526] Key type .llcrypt unregistered [ 4405.143106] Key type ._llcrypt unregistered [ 4412.098466] Key type ._llcrypt registered [ 4412.099600] Key type .llcrypt registered [ 4412.137293] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4412.450061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.456306] alg: No test for adler32 (adler32-zlib) [ 4413.313056] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4413.404279] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4414.975202] Key type lgssc registered [ 4415.399252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4417.897277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4417.901878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.040779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.422514] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4423.219508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.354224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4424.709248] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4427.008676] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4428.759130] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4430.911185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4431.014057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.805980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4435.062120] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4443.303475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4449.759875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4449.762451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.765949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.857072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4453.857117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.857812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4458.976476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4458.978982] Lustre: Skipped 2 previous similar messages [ 4463.071065] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4463.119332] Lustre: server umount lustre-OST0000 complete [ 4464.607726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4464.609622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.612691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.216418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.216712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.419169] Lustre: server umount lustre-MDT0000 complete [ 4471.570917] LustreError: 268026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314364 with bad export cookie 9592527274713167686 [ 4471.574094] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.575198] LustreError: 268026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4471.692901] Lustre: server umount lustre-MDT0001 complete [ 4475.052515] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4476.044980] Key type lgssc unregistered [ 4476.171460] LNet: 270990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4476.174617] LNetError: 270990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4476.185300] LNet: Removed LNI 192.168.206.115@tcp [ 4476.493603] Key type .llcrypt unregistered [ 4476.495296] Key type ._llcrypt unregistered [ 4484.936795] Key type ._llcrypt registered [ 4484.937709] Key type .llcrypt registered [ 4484.973529] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4485.227180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4485.237233] alg: No test for adler32 (adler32-zlib) [ 4486.083167] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4486.183406] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4487.767134] Key type lgssc registered [ 4488.143174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4490.742825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4490.749846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.880149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.153174] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4495.823145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.937635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4497.264032] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4499.522678] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4501.194094] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4503.313334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4503.417854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.276193] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4507.478490] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4516.772912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4526.047832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4526.050037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.055297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.073214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.074670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4527.075113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.671702] Lustre: server umount lustre-OST0000 complete [ 4536.799924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4536.802322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.807227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.809093] Lustre: Skipped 1 previous similar message [ 4539.100814] Lustre: server umount lustre-MDT0000 complete [ 4540.341106] LustreError: 272364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314433 with bad export cookie 7309110875024536202 [ 4540.344443] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.345216] LustreError: 272364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4540.455891] Lustre: server umount lustre-MDT0001 complete [ 4543.942426] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4544.993139] Key type lgssc unregistered [ 4545.130474] LNet: 276704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.133518] LNetError: 276704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.141440] LNet: Removed LNI 192.168.206.115@tcp [ 4545.476931] Key type .llcrypt unregistered [ 4545.478428] Key type ._llcrypt unregistered [ 4552.597897] Key type ._llcrypt registered [ 4552.599355] Key type .llcrypt registered [ 4552.637457] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4552.961728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4553.020462] alg: No test for adler32 (adler32-zlib) [ 4553.872854] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4553.971387] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4555.559143] Key type lgssc registered [ 4555.951497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4558.652513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4558.658794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.808512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.197179] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4564.016890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.150900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.609967] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4567.995214] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.717693] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4571.941287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4572.062220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4573.852513] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4576.211804] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4584.361485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4594.656832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4594.658192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.666812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4594.669906] Lustre: Skipped 1 previous similar message [ 4596.110544] Lustre: server umount lustre-OST0000 complete [ 4599.776434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.777405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.780259] Lustre: Skipped 2 previous similar messages [ 4603.548743] Lustre: server umount lustre-MDT0000 complete [ 4604.809194] LustreError: 280882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314497 with bad export cookie 9061754272420556627 [ 4604.812031] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.813911] LustreError: 280882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4604.934492] Lustre: server umount lustre-MDT0001 complete [ 4608.646723] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4609.707395] Key type lgssc unregistered [ 4609.839364] LNet: 281633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4609.842780] LNetError: 281633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4609.850302] LNet: Removed LNI 192.168.206.115@tcp [ 4610.154193] Key type .llcrypt unregistered [ 4610.155277] Key type ._llcrypt unregistered [ 4613.590424] Key type ._llcrypt registered [ 4613.591432] Key type .llcrypt registered [ 4613.626767] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4618.815660] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4624.749462] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4625.064885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4625.099963] alg: No test for adler32 (adler32-zlib) [ 4625.946554] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4626.028739] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4627.607110] Key type lgssc registered [ 4627.943217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4638.405689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4638.410642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.481204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4639.489292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4639.521546] Lustre: lustre-MDT0000: new disk, initializing [ 4639.542194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.547886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4640.857630] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4644.701889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.742888] Lustre: 285158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4644.767937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4644.769358] Lustre: Skipped 1 previous similar message [ 4644.802394] Lustre: lustre-MDT0001: new disk, initializing [ 4644.839357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4644.854240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4644.858458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4646.088117] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4649.705177] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4651.455451] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.523692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.609980] Lustre: lustre-OST0000: new disk, initializing [ 4653.611299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4653.613774] Lustre: 287122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4653.637528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.449980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4658.166505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4658.170347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4658.196714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4658.988556] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4660.349451] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4660.430346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4661.693670] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4661.763700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4663.265720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.266525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.268441] LustreError: Skipped 1 previous similar message [ 4663.273679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.276880] Lustre: Skipped 1 previous similar message [ 4664.461543] Lustre: server umount lustre-OST0000 complete [ 4668.384508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.385114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.388467] Lustre: Skipped 2 previous similar messages [ 4671.764664] Lustre: server umount lustre-MDT0000 complete [ 4672.962089] LustreError: 285148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314565 with bad export cookie 1230352280925317905 [ 4672.965236] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.965609] LustreError: 285148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4673.076526] Lustre: server umount lustre-MDT0001 complete [ 4675.760635] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:42:47 (1785314567) [ 4677.802310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.962797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4679.175367] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4681.860437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.966497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4683.208257] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4685.516696] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4687.224573] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4689.353630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.262607] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4693.446944] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4707.295767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4707.298125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.301947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4707.304490] Lustre: Skipped 1 previous similar message [ 4709.345862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4709.347036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4709.350996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.846124] Lustre: server umount lustre-OST0000 complete [ 4714.464018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.464574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4714.467915] Lustre: Skipped 1 previous similar message [ 4714.469354] Lustre: Skipped 1 previous similar message [ 4720.217726] Lustre: server umount lustre-MDT0000 complete [ 4721.475100] LustreError: 289508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314614 with bad export cookie 1230352280925318773 [ 4721.476726] LustreError: 290264:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4721.477836] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4721.479431] LustreError: 289508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4721.594392] Lustre: server umount lustre-MDT0001 complete [ 4725.127996] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4726.123814] Key type lgssc unregistered [ 4726.249329] LNet: 293081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4726.251288] LNetError: 293081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4726.259265] LNet: Removed LNI 192.168.206.115@tcp [ 4726.559499] Key type .llcrypt unregistered [ 4726.561127] Key type ._llcrypt unregistered [ 4734.773419] Key type ._llcrypt registered [ 4734.774595] Key type .llcrypt registered [ 4734.809593] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4740.724670] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4741.057662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4741.129653] alg: No test for adler32 (adler32-zlib) [ 4741.971154] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4742.054358] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4743.623137] Key type lgssc registered [ 4743.988527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4758.546824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4758.551352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.627404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4759.636624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4759.670836] Lustre: lustre-MDT0000: new disk, initializing [ 4759.694864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.701787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4761.015745] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4765.081082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.122970] Lustre: 296878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4765.145360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4765.147903] Lustre: Skipped 1 previous similar message [ 4765.183047] Lustre: lustre-MDT0001: new disk, initializing [ 4765.213381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4765.224961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4765.228477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4766.622904] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4770.374471] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4772.301564] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4774.633228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.731462] Lustre: lustre-OST2710: new disk, initializing [ 4774.733744] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4774.737464] Lustre: 298843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4774.769514] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4775.672560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4775.675966] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4775.715190] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4776.727812] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4780.392969] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4782.553336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4782.613671] Lustre: lustre-OST03e8: new disk, initializing [ 4782.616059] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4782.619665] Lustre: 299906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4782.657256] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4784.380804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4784.383810] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4784.408270] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4784.452687] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4786.796655] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4789.461646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4789.528265] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4790.749750] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4790.811969] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4792.043231] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4792.106805] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4793.306917] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4793.367803] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4794.849474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.851177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4794.855906] Lustre: Skipped 3 previous similar messages [ 4794.858122] Lustre: Skipped 2 previous similar messages [ 4799.969280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.971692] Lustre: Skipped 2 previous similar messages [ 4800.423727] Lustre: server umount lustre-MDT0000 complete [ 4801.698181] LustreError: 296869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314694 with bad export cookie 13742682035633673310 [ 4801.702522] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.703590] LustreError: 296869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4801.825126] Lustre: server umount lustre-MDT0001 complete [ 4813.322640] Lustre: server umount lustre-OST2710 complete [ 4824.854563] Lustre: server umount lustre-OST03e8 complete [ 4830.695445] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4833.883785] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4845.409494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.488109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4845.495742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4845.532783] Lustre: lustre-MDT0000: new disk, initializing [ 4845.553302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.560685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4846.819836] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4850.937963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.975719] Lustre: 305144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4851.006251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4851.083280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4852.365064] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4856.080966] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.924718] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4860.096812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.183526] Lustre: lustre-OST0000: new disk, initializing [ 4860.185648] Lustre: Skipped 1 previous similar message [ 4860.187861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4860.190043] Lustre: Skipped 2 previous similar messages [ 4860.192152] Lustre: 307114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4860.224215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4860.226559] Lustre: Skipped 1 previous similar message [ 4860.917968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4860.920847] Lustre: Skipped 1 previous similar message [ 4860.922704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4860.951729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4862.059933] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4865.651517] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4867.048797] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4867.125447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4868.484946] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4868.554477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4871.138425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4871.138477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.140867] LustreError: Skipped 1 previous similar message [ 4871.145758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4871.148701] Lustre: Skipped 1 previous similar message [ 4875.278043] Lustre: server umount lustre-OST0000 complete [ 4876.767756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.770606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.774873] Lustre: Skipped 1 previous similar message [ 4876.776869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4881.376316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.717756] Lustre: server umount lustre-MDT0000 complete [ 4883.934158] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314776 with bad export cookie 13742682035633674976 [ 4883.937403] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.939608] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4884.056532] Lustre: server umount lustre-MDT0001 complete [ 4886.684102] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:46:18 (1785314778) [ 4887.179127] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4887.683850] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:46:19 (1785314779) [ 4888.598513] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4897.192688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.272597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4897.281670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4897.340949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.712554] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4901.554929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.588155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4901.597612] Lustre: 310963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4903.073759] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4918.881590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.941453] Lustre: Denying initial registration attempt from nid 192.168.206.115@tcp, specified as failover [ 4918.943853] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4918.946183] LustreError: 312415:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4918.948047] LustreError: 312415:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4918.949631] LustreError: 312415:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4918.967997] Lustre: server umount lustre-OST0000 complete [ 4918.969221] LustreError: 312415:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4922.335741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4922.337883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.341629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.343575] Lustre: Skipped 2 previous similar messages [ 4927.117895] Lustre: server umount lustre-MDT0000 complete [ 4928.252267] LustreError: 311703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314821 with bad export cookie 13742682035633675830 [ 4928.255151] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4928.255502] LustreError: 311703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4928.353414] Lustre: server umount lustre-MDT0001 complete [ 4931.777435] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4932.736375] Key type lgssc unregistered [ 4932.869430] LNet: 313534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4932.872356] LNetError: 313534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4932.883341] LNet: Removed LNI 192.168.206.115@tcp [ 4933.192866] Key type .llcrypt unregistered [ 4933.194284] Key type ._llcrypt unregistered [ 4936.568629] Key type ._llcrypt registered [ 4936.569447] Key type .llcrypt registered [ 4936.603078] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4936.942185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4936.966223] alg: No test for adler32 (adler32-zlib) [ 4937.816878] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4937.896215] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4939.471138] Key type lgssc registered [ 4939.790102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4962.994166] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4965.849944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4965.855865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.930816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4966.937530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4966.985823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.216230] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4970.913488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.943409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4970.952214] Lustre: 316272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4971.035876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4972.307172] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4974.672641] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4976.558381] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4991.841716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4991.907321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4991.975171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4993.962087] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4996.491764] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4997.225928] Lustre: server umount lustre-OST0000 complete [ 4998.639236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4998.640416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4998.643846] Lustre: Skipped 1 previous similar message [ 5003.744699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5003.746687] Lustre: Skipped 2 previous similar messages [ 5004.502763] Lustre: server umount lustre-MDT0000 complete [ 5005.760224] LustreError: 316263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314898 with bad export cookie 4496084482178474874 [ 5005.763652] LustreError: 316263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5005.763731] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.883454] Lustre: server umount lustre-MDT0001 complete [ 5009.681878] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5010.709811] Key type lgssc unregistered [ 5010.828371] LNet: 319837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5010.830468] LNetError: 319837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5010.837294] LNet: Removed LNI 192.168.206.115@tcp [ 5011.154812] Key type .llcrypt unregistered [ 5011.155819] Key type ._llcrypt unregistered [ 5020.315770] Key type ._llcrypt registered [ 5020.316857] Key type .llcrypt registered [ 5020.353192] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5020.656057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5020.662246] alg: No test for adler32 (adler32-zlib) [ 5021.510102] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5021.587451] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5023.167084] Key type lgssc registered [ 5023.498088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5025.847310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5025.853515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.998128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.305351] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5031.151727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.283479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.577807] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5034.841797] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5036.487033] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5040.482257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5040.595628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.429959] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5044.754615] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5060.350304] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 5062.114102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.115368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.118413] Lustre: Skipped 2 previous similar messages [ 5062.120034] Lustre: Skipped 1 previous similar message [ 5062.123145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5063.385072] Lustre: server umount lustre-MDT0000 complete [ 5064.600016] LustreError: 321224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314957 with bad export cookie 11527449383673628260 [ 5064.603606] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5064.604771] LustreError: 321224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5065.464303] LustreError: 321237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.470303] LustreError: 321237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5065.470413] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 5065.475189] Lustre: Skipped 1 previous similar message [ 5067.231781] LustreError: 321237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.231979] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.279672] LustreError: 321971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.279928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5069.286786] Lustre: Skipped 1 previous similar message [ 5070.764688] Lustre: server umount lustre-MDT0001 complete [ 5072.853746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5075.693755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.759601] LustreError: 320554:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000071d04817 x1872038371287168/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5089.882427] LustreError: 323072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.888605] LustreError: 323072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5089.908267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.935968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5091.240806] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5093.953273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.070111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5095.077508] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5095.358993] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5097.677070] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5099.173737] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5102.234304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5103.926557] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5104.409056] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5109.729563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.734478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.734830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.739818] Lustre: Skipped 2 previous similar messages [ 5111.761091] Lustre: server umount lustre-OST0000 complete [ 5114.849050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.849182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5114.852711] Lustre: Skipped 1 previous similar message [ 5119.127066] Lustre: server umount lustre-MDT0000 complete [ 5119.968100] LustreError: 324856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.975052] LustreError: 324856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5120.446055] LustreError: 321267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315013 with bad export cookie 11527449383673635414 [ 5120.448646] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.450510] LustreError: 321267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5120.561753] Lustre: server umount lustre-MDT0001 complete [ 5124.005496] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5124.926866] Key type lgssc unregistered [ 5125.045374] LNet: 327525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5125.048422] LNetError: 327525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5125.056259] LNet: Removed LNI 192.168.206.115@tcp [ 5125.349500] Key type .llcrypt unregistered [ 5125.350572] Key type ._llcrypt unregistered [ 5134.170215] Key type ._llcrypt registered [ 5134.171346] Key type .llcrypt registered [ 5134.208451] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5134.618869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5134.624844] alg: No test for adler32 (adler32-zlib) [ 5135.478305] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5135.578975] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5137.159157] Key type lgssc registered [ 5137.553789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5140.157016] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5140.163682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.307729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.706751] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5145.382163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.500727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5146.761658] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5148.936192] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5150.832632] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5155.807870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5155.811120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.815327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.832324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.833127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.040557] Lustre: server umount lustre-MDT0000 complete [ 5159.286369] LustreError: 328899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315052 with bad export cookie 1918445525765314744 [ 5159.287322] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.289603] LustreError: 328899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5159.411524] Lustre: server umount lustre-MDT0001 complete [ 5164.105336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.197041] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5164.206872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5164.283410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.593472] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5168.453135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.484080] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5168.493708] Lustre: 331503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5168.581864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5169.799604] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5172.140707] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5173.974836] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5178.530396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.601280] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5178.605106] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5178.608866] LustreError: 333385:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5178.611096] LustreError: 333385:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5178.613332] LustreError: 333385:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5178.636638] Lustre: server umount lustre-OST0000 complete [ 5178.637871] LustreError: 333385:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5183.523025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.606530] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5183.677143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.602162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5187.889360] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5190.322531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5190.350944] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5190.374275] Lustre: lustre-OST0001: new disk, initializing [ 5190.375908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5190.378117] Lustre: 334915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5191.430427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5191.433383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5191.459089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5191.481637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5192.127191] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5194.285073] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5196.770957] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.771089] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.775017] LustreError: Skipped 1 previous similar message [ 5196.779704] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5196.784179] Lustre: Skipped 2 previous similar messages [ 5197.246757] Lustre: server umount lustre-OST0001 complete [ 5201.889947] LustreError: 333851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.889967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.891250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5201.891437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5201.894343] LustreError: 333851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5201.899796] Lustre: Skipped 2 previous similar messages [ 5204.623813] Lustre: server umount lustre-OST0000 complete [ 5207.009096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.010349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5207.012406] Lustre: Skipped 1 previous similar message [ 5207.014388] Lustre: Skipped 1 previous similar message [ 5217.248613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.250939] Lustre: Skipped 3 previous similar messages [ 5220.319124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5220.378924] Lustre: server umount lustre-MDT0000 complete [ 5221.345341] LustreError: 331514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.355131] LustreError: 331514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5221.676418] LustreError: 331494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315114 with bad export cookie 1918445525765315283 [ 5221.679313] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.680870] LustreError: 331494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5221.796239] Lustre: server umount lustre-MDT0001 complete [ 5225.540160] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5226.560768] Key type lgssc unregistered [ 5226.694416] LNet: 336875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5226.696892] LNetError: 336875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5226.708316] LNet: Removed LNI 192.168.206.115@tcp [ 5227.007912] Key type .llcrypt unregistered [ 5227.008883] Key type ._llcrypt unregistered [ 5256.802879] Key type ._llcrypt registered [ 5256.803933] Key type .llcrypt registered [ 5256.841892] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5262.990683] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5263.435758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5263.443466] alg: No test for adler32 (adler32-zlib) [ 5264.293588] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5264.381255] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5265.959152] Key type lgssc registered [ 5266.323828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5277.083423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5277.087855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.163166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5278.172316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5278.207559] Lustre: lustre-MDT0000: new disk, initializing [ 5278.225932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.231504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5279.445102] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5283.412993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5283.453101] Lustre: 341777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5283.474742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5283.476602] Lustre: Skipped 1 previous similar message [ 5283.510472] Lustre: lustre-MDT0001: new disk, initializing [ 5283.541779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5283.552550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5283.556223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5284.736369] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5288.288221] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5289.993165] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5292.127404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.251304] Lustre: lustre-OST0000: new disk, initializing [ 5292.253735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5292.256772] Lustre: 343745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5292.290708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.100711] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5297.878845] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5299.325304] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5301.748750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5301.752924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5301.782553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5302.433681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5303.715984] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5303.778557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5306.851183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5306.852979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.853028] LustreError: Skipped 1 previous similar message [ 5306.856158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5306.858356] Lustre: Skipped 1 previous similar message [ 5310.491829] Lustre: server umount lustre-OST0000 complete [ 5311.968864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.969773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5311.971975] Lustre: Skipped 2 previous similar messages [ 5317.088066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5317.089944] Lustre: Skipped 2 previous similar messages [ 5322.208067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.209796] Lustre: Skipped 1 previous similar message [ 5326.303200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5326.308123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.311994] Lustre: Skipped 1 previous similar message [ 5326.358057] LustreError: 345303:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003ccb8486) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5328.437420] Lustre: server umount lustre-MDT0000 complete [ 5329.901215] LustreError: 341769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315222 with bad export cookie 3746024828258995328 [ 5329.903952] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5329.906081] LustreError: 341769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5330.036843] Lustre: server umount lustre-MDT0001 complete [ 5333.083260] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:53:45 (1785315225) [ 5335.765046] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:53:47 (1785315227) [ 5338.803760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.008887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5340.390966] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5343.305576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.433965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5344.820991] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5347.319884] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5349.194398] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5351.535943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5351.659500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.595585] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5355.922476] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.966334] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5370.565516] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5371.177558] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5371.807076] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5372.497783] Lustre: Failing over lustre-MDT0000 [ 5372.645928] Lustre: server umount lustre-MDT0000 complete [ 5374.435876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.436387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5374.441725] LustreError: 346520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.449575] LustreError: 346520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5376.769182] LustreError: 347950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.553575] LustreError: 346519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.557884] LustreError: 346519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5381.880236] LustreError: 346515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.737126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.775994] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.861411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.885320] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5387.000784] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5387.171594] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5391.338785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5391.346356] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5391.366248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5393.123429] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.641910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5395.512681] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5396.151155] Lustre: Failing over lustre-MDT0000 [ 5396.304102] Lustre: server umount lustre-MDT0000 complete [ 5396.449354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.450625] LustreError: 346519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.454220] Lustre: Skipped 4 previous similar messages [ 5396.462117] LustreError: 346519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5406.688369] LustreError: 347245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5406.694930] LustreError: 347245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5409.144526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.185266] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5409.288014] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5410.580493] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5412.600502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5414.374770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5414.376911] Lustre: Skipped 2 previous similar messages [ 5414.385963] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5414.402706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5416.183737] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5416.735738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5419.489091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.490346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5419.491020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5419.493767] Lustre: Skipped 1 previous similar message [ 5425.107298] Lustre: server umount lustre-MDT0000 complete [ 5426.291862] LustreError: 349311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315319 with bad export cookie 3746024828258997995 [ 5426.295425] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.295842] LustreError: 349311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5426.400810] Lustre: server umount lustre-MDT0001 complete [ 5437.918476] Lustre: server umount lustre-OST0000 complete [ 5442.924719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.065212] LustreError: 352869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.071517] LustreError: 352869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5443.088568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.090755] Lustre: Skipped 1 previous similar message [ 5444.420640] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5447.241383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.681267] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5451.008874] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5452.774900] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5454.908994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.805583] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5459.139459] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5467.305612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5472.737800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.737852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.741190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5472.744497] Lustre: Skipped 2 previous similar messages [ 5472.749240] Lustre: Skipped 6 previous similar messages [ 5476.746776] Lustre: server umount lustre-OST0000 complete [ 5478.367952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5492.191159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5492.245477] Lustre: server umount lustre-MDT0000 complete [ 5493.518972] LustreError: 352851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315386 with bad export cookie 3746024828258998716 [ 5493.522657] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.523763] LustreError: 352851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5493.631973] Lustre: server umount lustre-MDT0001 complete [ 5497.344528] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5498.355348] Key type lgssc unregistered [ 5498.478357] LNet: 356419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5498.480532] LNetError: 356419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5498.487282] LNet: Removed LNI 192.168.206.115@tcp [ 5498.791244] Key type .llcrypt unregistered [ 5498.792173] Key type ._llcrypt unregistered [ 5508.407695] Key type ._llcrypt registered [ 5508.408532] Key type .llcrypt registered [ 5508.440162] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5508.864635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5508.872389] alg: No test for adler32 (adler32-zlib) [ 5509.724099] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5509.809553] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5511.383109] Key type lgssc registered [ 5511.734202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5514.220195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5514.227352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.366587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.701467] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5519.507141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.632882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5520.950930] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5523.330533] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5525.135695] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5527.371830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.478085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.279274] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5531.652958] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5540.773790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5550.560518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5550.563868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.569730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.076884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.076989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.078945] Lustre: Skipped 1 previous similar message [ 5551.084027] Lustre: Skipped 1 previous similar message [ 5552.219679] Lustre: server umount lustre-MDT0000 complete [ 5553.474473] LustreError: 358645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315446 with bad export cookie 11231237106002136038 [ 5553.477899] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5553.479522] LustreError: 358645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5553.587163] Lustre: server umount lustre-MDT0001 complete [ 5565.075359] Lustre: server umount lustre-OST0000 complete [ 5569.429996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.567849] LustreError: 361641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.575650] LustreError: 361641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5569.588390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.859410] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5573.600906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.721647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5575.032764] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5577.355801] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5579.148675] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.378341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.407765] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5585.878264] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.786704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5604.320039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.322769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.327395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.329900] Lustre: Skipped 1 previous similar message [ 5609.365726] Lustre: server umount lustre-MDT0000 complete [ 5610.464100] LustreError: 361642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.469628] LustreError: 361642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5610.579137] LustreError: 361623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315503 with bad export cookie 11231237106002137137 [ 5610.582498] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.586220] LustreError: 361623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5610.703112] Lustre: server umount lustre-MDT0001 complete [ 5622.165800] Lustre: server umount lustre-OST0000 complete [ 5626.966097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.115553] LustreError: 365569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.136931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.139924] Lustre: Skipped 1 previous similar message [ 5628.466588] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5631.144451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.501108] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5634.915539] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.684990] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.906650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.710637] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5642.908441] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5652.325472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5657.568474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5657.569238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.571536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.575102] Lustre: Skipped 3 previous similar messages [ 5657.579790] Lustre: Skipped 4 previous similar messages [ 5660.432201] Lustre: server umount lustre-OST0000 complete [ 5662.175730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5662.177882] LustreError: Skipped 1 previous similar message [ 5662.179088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.182758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5672.928698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5672.931583] Lustre: Skipped 4 previous similar messages [ 5675.999102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5676.057652] Lustre: server umount lustre-MDT0000 complete [ 5677.291126] LustreError: 365549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315570 with bad export cookie 11231237106002138145 [ 5677.294152] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.298067] LustreError: 365549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5677.418131] Lustre: server umount lustre-MDT0001 complete [ 5681.035535] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5682.166338] Key type lgssc unregistered [ 5682.296339] LNet: 369122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.298738] LNetError: 369122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.305295] LNet: Removed LNI 192.168.206.115@tcp [ 5682.639905] Key type .llcrypt unregistered [ 5682.640938] Key type ._llcrypt unregistered [ 5692.072701] Key type ._llcrypt registered [ 5692.074085] Key type .llcrypt registered [ 5692.108101] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5692.417043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5692.422215] alg: No test for adler32 (adler32-zlib) [ 5693.265509] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5693.341424] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5694.911122] Key type lgssc registered [ 5695.246093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5697.606624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5697.612154] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.615053] LustreError: 370552:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5697.622989] LustreError: 370552:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5697.625309] LustreError: 370552:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5697.627677] LustreError: 370552:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5697.630245] LustreError: 370552:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5700.087171] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5700.089704] LustreError: 370948:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5700.097912] LustreError: 370948:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5700.100692] LustreError: 370948:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5700.103462] LustreError: 370948:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5700.106329] LustreError: 370948:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5706.293165] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5707.207870] Key type lgssc unregistered [ 5707.320340] LNet: 371948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5707.322425] LNetError: 371948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5707.332250] LNet: Removed LNI 192.168.206.115@tcp [ 5707.603223] Key type .llcrypt unregistered [ 5707.604116] Key type ._llcrypt unregistered [ 5713.049181] Key type ._llcrypt registered [ 5713.050622] Key type .llcrypt registered [ 5713.086845] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5718.692294] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5719.003913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.052741] alg: No test for adler32 (adler32-zlib) [ 5719.923469] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5720.019229] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5721.599140] Key type lgssc registered [ 5721.957790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.163632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.168966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.249282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.258485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.298072] Lustre: lustre-MDT0000: new disk, initializing [ 5734.319893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.325702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5735.638278] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5739.657807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.698378] Lustre: 375076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5739.719725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5739.721476] Lustre: Skipped 1 previous similar message [ 5739.757585] Lustre: lustre-MDT0001: new disk, initializing [ 5739.790093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.804116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5739.807340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5741.102722] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5744.728123] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5746.542204] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5748.699103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.808631] Lustre: lustre-OST0000: new disk, initializing [ 5748.811418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5748.815884] Lustre: 377046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5748.843939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.010045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5750.013799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.047336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5750.692545] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5754.262739] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5755.612569] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5755.677442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5757.092753] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5757.162260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5759.821753] Lustre: server umount lustre-OST0000 complete [ 5765.600868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5765.600913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.601666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5765.613389] Lustre: Skipped 1 previous similar message [ 5770.720454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.723015] Lustre: Skipped 2 previous similar messages [ 5775.327108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5775.387784] Lustre: server umount lustre-MDT0000 complete [ 5776.647133] LustreError: 375069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315669 with bad export cookie 16690065917082067293 [ 5776.650522] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.652168] LustreError: 375069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5787.151107] Lustre: server umount lustre-MDT0001 complete [ 5789.877305] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5790.468633] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:01:22 (1785315682) [ 5792.645783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.803499] LustreError: 379513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.808355] LustreError: 379513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5792.826156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.173550] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5796.963426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.081319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5798.368407] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5800.714572] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5802.487102] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5804.756328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5806.735873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5809.143606] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5811.427581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.489235] Lustre: lustre-OST0001: new disk, initializing [ 5811.490671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5811.493720] Lustre: 382349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5811.522633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5811.525600] Lustre: Skipped 1 previous similar message [ 5813.463674] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5817.244097] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5818.873842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5818.876853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5818.903599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5823.971316] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.971938] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.973458] LustreError: Skipped 1 previous similar message [ 5823.974079] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5823.977185] Lustre: Skipped 1 previous similar message [ 5825.474905] Lustre: server umount lustre-OST0001 complete [ 5826.148769] LustreError: 383070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.091984] LustreError: 381332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.098245] LustreError: 381332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5834.212375] LustreError: 381334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.217347] LustreError: 381334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5837.791839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5837.794963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5837.798568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5837.800691] Lustre: Skipped 1 previous similar message [ 5839.331184] LustreError: 381335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.335511] LustreError: 381335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5843.597816] Lustre: server umount lustre-OST0000 complete [ 5848.543923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5848.546150] LustreError: Skipped 1 previous similar message [ 5848.547304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.550510] Lustre: Skipped 1 previous similar message [ 5848.552232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.553962] Lustre: Skipped 3 previous similar messages [ 5859.295098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5859.353827] Lustre: server umount lustre-MDT0000 complete [ 5859.808094] LustreError: 381811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.812638] LustreError: 381811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5860.566426] LustreError: 379494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315753 with bad export cookie 16690065917082068161 [ 5860.568847] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5860.569731] LustreError: 379494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5866.749765] Lustre: server umount lustre-MDT0001 complete [ 5870.493885] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5871.504778] Key type lgssc unregistered [ 5871.628359] LNet: 384333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5871.630512] LNetError: 384333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5871.638313] LNet: Removed LNI 192.168.206.115@tcp [ 5871.934719] Key type .llcrypt unregistered [ 5871.935699] Key type ._llcrypt unregistered [ 5889.574294] Key type ._llcrypt registered [ 5889.575319] Key type .llcrypt registered [ 5889.615400] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5889.940385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5889.950248] alg: No test for adler32 (adler32-zlib) [ 5890.807513] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5890.900887] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5892.479105] Key type lgssc registered [ 5892.853604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5895.432159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5895.437708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.520623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5896.528832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5896.591161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.967445] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5900.819054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.850366] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5900.861023] Lustre: 387122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5900.953585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5902.323188] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5904.746420] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5906.478250] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5911.520765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5911.520804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.521407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.528935] Lustre: Skipped 1 previous similar message [ 5913.239454] Lustre: server umount lustre-MDT0000 complete [ 5914.407108] LustreError: 387157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315807 with bad export cookie 13341467874631191609 [ 5914.410280] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.412607] LustreError: 387157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5914.520153] Lustre: server umount lustre-MDT0001 complete [ 5917.547994] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:03:29 (1785315809) [ 5919.722754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.898756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5921.214462] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5923.930942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.044563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5925.477135] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5927.979102] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.862395] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5932.075802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5932.144974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5932.213467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.139281] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5936.439529] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5945.463698] Lustre: Permanently deactivating lustre-OST0000 [ 5945.469052] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5947.950481] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5948.641659] LustreError: 392520:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5948.647467] LustreError: 392520:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5948.650979] LustreError: 392520:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5949.258298] LustreError: 392568:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5949.260541] LustreError: 392568:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5956.050800] Lustre: server umount lustre-OST0000 complete [ 5956.949427] LustreError: 392817:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5956.951593] LustreError: 392817:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5959.136708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.137574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.140484] Lustre: Skipped 1 previous similar message [ 5959.143238] Lustre: Skipped 1 previous similar message [ 5964.256095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.258188] Lustre: Skipped 2 previous similar messages [ 5967.350966] Lustre: server umount lustre-MDT0000 complete [ 5968.594154] LustreError: 391464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315861 with bad export cookie 13341467874631192176 [ 5968.597678] LustreError: 391464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5968.598783] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5979.151822] Lustre: server umount lustre-MDT0001 complete [ 5981.996592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.403505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5984.339299] LustreError: 393974:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5986.470874] Lustre: 394312:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.115@tcp [ 5987.154813] Lustre: server umount MGS complete [ 5990.910688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.056866] LustreError: 395036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.061439] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5991.061673] LustreError: 395036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5991.063692] Lustre: Skipped 1 previous similar message [ 5991.080580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5992.279988] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5994.879910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.974979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5996.410548] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5998.789350] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6000.612407] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6001.229481] Lustre: Permanently reactivating lustre-OST0000 [ 6001.234923] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6001.237711] Lustre: Skipped 2 previous similar messages [ 6025.183136] LustreError: 396766:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6027.972826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6028.084074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.085716] Lustre: Skipped 1 previous similar message [ 6029.852833] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6032.115374] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6045.664457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6045.665360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.666467] LustreError: Skipped 1 previous similar message [ 6045.671817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.673801] Lustre: Skipped 1 previous similar message [ 6050.187069] Lustre: server umount lustre-OST0000 complete [ 6055.904473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.905191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.909222] Lustre: Skipped 2 previous similar messages [ 6055.913017] Lustre: Skipped 1 previous similar message [ 6057.558484] Lustre: server umount lustre-MDT0000 complete [ 6058.679342] LustreError: 396607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315951 with bad export cookie 13341467874631193198 [ 6058.682175] LustreError: 396607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6058.682389] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6058.790357] Lustre: server umount lustre-MDT0001 complete [ 6062.357196] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6063.277760] Key type lgssc unregistered [ 6063.393346] LNet: 399657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6063.396367] LNetError: 399657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6063.404240] LNet: Removed LNI 192.168.206.115@tcp [ 6063.687075] Key type .llcrypt unregistered [ 6063.687914] Key type ._llcrypt unregistered [ 6069.215950] Key type ._llcrypt registered [ 6069.217052] Key type .llcrypt registered [ 6069.249448] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6075.537262] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6075.945794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6075.974036] alg: No test for adler32 (adler32-zlib) [ 6076.820250] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6076.907860] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6078.479108] Key type lgssc registered [ 6078.804596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6090.531570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6090.536904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.632360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6091.643546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6091.684486] Lustre: lustre-MDT0000: new disk, initializing [ 6091.705850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.713275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6093.098102] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6097.048614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.091875] Lustre: 402794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6097.114966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6097.117272] Lustre: Skipped 1 previous similar message [ 6097.155475] Lustre: lustre-MDT0001: new disk, initializing [ 6097.188933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6097.201515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6097.205147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6098.455137] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6102.023937] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6103.863503] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6105.938952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.031786] Lustre: lustre-OST0000: new disk, initializing [ 6106.033497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6106.035733] Lustre: 404761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6106.063965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.839397] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6109.493556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6109.497688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6109.526820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6111.331860] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6112.683310] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6112.758027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6114.027711] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6114.096333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6114.784478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.785755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.786844] LustreError: Skipped 1 previous similar message [ 6114.786975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.790595] Lustre: Skipped 1 previous similar message [ 6119.907458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.909484] Lustre: Skipped 2 previous similar messages [ 6120.787968] Lustre: server umount lustre-OST0000 complete [ 6122.975931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6122.978916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.983719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.024551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.025243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.144019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.146243] Lustre: Skipped 2 previous similar messages [ 6136.287102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6136.355778] Lustre: server umount lustre-MDT0000 complete [ 6137.656061] LustreError: 403588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316030 with bad export cookie 1171945907340118597 [ 6137.658916] LustreError: 403588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6137.659861] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.767502] Lustre: server umount lustre-MDT0001 complete [ 6140.525280] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6141.071185] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:07:13 (1785316033) [ 6143.239228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.402936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.700791] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6147.344667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6147.455994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.720918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6150.991047] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6152.782950] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6154.927804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.877846] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6159.271223] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6159.864904] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6170.440311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6173.664233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.664996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.666671] LustreError: Skipped 1 previous similar message [ 6173.672068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.674962] Lustre: Skipped 3 previous similar messages [ 6177.926590] Lustre: server umount lustre-OST0000 complete [ 6183.391908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6183.394194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.397427] Lustre: Skipped 1 previous similar message [ 6185.363586] Lustre: server umount lustre-MDT0000 complete [ 6186.637377] LustreError: 407196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316079 with bad export cookie 1171945907340119479 [ 6186.640974] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.641927] LustreError: 407196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6186.751844] Lustre: server umount lustre-MDT0001 complete [ 6190.427463] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6191.444959] Key type lgssc unregistered [ 6191.577419] LNet: 410959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6191.580499] LNetError: 410959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6191.591328] LNet: Removed LNI 192.168.206.115@tcp [ 6191.891829] Key type .llcrypt unregistered [ 6191.892845] Key type ._llcrypt unregistered [ 6199.561210] Key type ._llcrypt registered [ 6199.562655] Key type .llcrypt registered [ 6199.599284] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6199.693232] Key type .llcrypt unregistered [ 6199.694240] Key type ._llcrypt unregistered [ 6206.832634] Key type ._llcrypt registered [ 6206.834244] Key type .llcrypt registered [ 6206.870426] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6207.191236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6207.240053] alg: No test for adler32 (adler32-zlib) [ 6208.090711] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6208.177490] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6209.759131] Key type lgssc registered [ 6210.125472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6212.663472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6212.670608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.815650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.194733] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6218.026451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.149655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.087654] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6222.543662] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.265357] LustreError: 413145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.269745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6223.271912] LustreError: 413145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6224.862142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.991929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.351424] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6233.568886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.571994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.577611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6235.104605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.106915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.110642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.696319] Lustre: server umount lustre-OST0000 complete [ 6243.811618] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6243.817653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.827706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.830522] Lustre: Skipped 1 previous similar message [ 6245.856291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6245.856943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.075391] Lustre: server umount lustre-MDT0000 complete [ 6248.383394] LustreError: 413126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316141 with bad export cookie 3280587018876751218 [ 6248.386046] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.386856] LustreError: 413126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6248.506490] Lustre: server umount lustre-MDT0001 complete [ 6252.130539] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6253.171064] Key type lgssc unregistered [ 6253.307494] LNet: 416290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6253.310170] LNetError: 416290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6253.318416] LNet: Removed LNI 192.168.206.115@tcp [ 6253.626785] Key type .llcrypt unregistered [ 6253.628322] Key type ._llcrypt unregistered [ 6263.067856] Key type ._llcrypt registered [ 6263.068783] Key type .llcrypt registered [ 6263.104551] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6263.576781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6263.588027] alg: No test for adler32 (adler32-zlib) [ 6264.443770] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6264.546185] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6266.127098] Key type lgssc registered [ 6266.524541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6269.099884] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6269.106369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.511338] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6289.631213] LustreError: 417657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 6316.255205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.205859] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6320.802129] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6323.108446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.309961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.338773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6324.801697] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6327.648778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.778507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6329.199917] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6331.655294] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.535887] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6335.969205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.973677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6335.975348] Lustre: Skipped 1 previous similar message [ 6341.090480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6341.094492] Lustre: Skipped 1 previous similar message [ 6342.037790] Lustre: server umount lustre-OST0000 complete [ 6346.209931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.210816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6346.214133] Lustre: Skipped 2 previous similar messages [ 6346.217475] Lustre: Skipped 1 previous similar message [ 6349.467270] Lustre: server umount lustre-MDT0000 complete [ 6350.743177] LustreError: 418724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316243 with bad export cookie 3580633341433403034 [ 6350.746807] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.746923] LustreError: 418724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6350.872942] Lustre: server umount lustre-MDT0001 complete [ 6354.657207] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6355.719139] Key type lgssc unregistered [ 6355.849462] LNet: 421226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6355.853630] LNetError: 421226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6355.860420] LNet: Removed LNI 192.168.206.115@tcp [ 6356.178390] Key type .llcrypt unregistered [ 6356.180088] Key type ._llcrypt unregistered [ 6365.174229] Key type ._llcrypt registered [ 6365.175474] Key type .llcrypt registered [ 6365.206720] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6365.553790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6365.570296] alg: No test for adler32 (adler32-zlib) [ 6366.410954] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6366.501905] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6368.087134] Key type lgssc registered [ 6368.405170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6370.784446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6370.790182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.920401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.210993] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6375.936492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.046781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6377.367588] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6379.586566] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.263186] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6383.292649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.390463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.126571] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6387.391501] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6396.709086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6401.503940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6401.506339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.510937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.018323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.018598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.024245] Lustre: Skipped 1 previous similar message [ 6402.026848] Lustre: Skipped 1 previous similar message [ 6404.637903] Lustre: server umount lustre-MDT0000 complete [ 6405.383798] LustreError: 424461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.15@tcp arrived at 1785316298 with bad export cookie 12666981425560625959 [ 6405.388907] LustreError: 424461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6406.005511] LustreError: 422600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316298 with bad export cookie 12666981425560625308 [ 6406.008967] LustreError: 422600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6406.009176] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.111974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6406.114150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.118423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.121442] Lustre: Skipped 2 previous similar messages [ 6407.136397] LustreError: 422638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.142328] LustreError: 422638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6411.167740] LustreError: 422620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.168092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.172762] LustreError: 422620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6411.174915] Lustre: Skipped 1 previous similar message [ 6412.114200] Lustre: server umount lustre-OST0000 complete [ 6414.322218] Lustre: server umount lustre-MDT0001 complete [ 6417.735245] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6418.655314] Key type lgssc unregistered [ 6418.764376] LNet: 426322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6418.767140] LNetError: 426322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6418.774266] LNet: Removed LNI 192.168.206.115@tcp [ 6419.062758] Key type .llcrypt unregistered [ 6419.063705] Key type ._llcrypt unregistered [ 6427.722575] Key type ._llcrypt registered [ 6427.724296] Key type .llcrypt registered [ 6427.763323] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6428.190269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6428.257788] alg: No test for adler32 (adler32-zlib) [ 6429.110667] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6429.208700] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6430.791215] Key type lgssc registered [ 6431.153194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6433.563979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6433.570102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.688618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.894528] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6438.714358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.838249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6440.197646] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6442.741584] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6444.619249] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6446.889550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.995229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.935486] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6451.427905] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6459.302661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6464.482641] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6464.482987] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.489724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6464.491914] Lustre: Skipped 1 previous similar message [ 6467.330418] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 6469.264096] Lustre: server umount lustre-MDT0001 complete [ 6469.601442] LustreError: 427714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6469.606881] LustreError: 427714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6474.720729] LustreError: 427733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.721791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.729709] Lustre: Skipped 1 previous similar message [ 6474.731708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.732032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6477.071822] Lustre: server umount lustre-OST0000 complete [ 6478.410463] Lustre: server umount lustre-MDT0000 complete [ 6483.078273] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6484.130278] Key type lgssc unregistered [ 6484.265438] LNet: 431414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6484.268408] LNetError: 431414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6484.278338] LNet: Removed LNI 192.168.206.115@tcp [ 6484.606703] Key type .llcrypt unregistered [ 6484.607773] Key type ._llcrypt unregistered [ 6492.828272] Key type ._llcrypt registered [ 6492.829368] Key type .llcrypt registered [ 6492.867955] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6499.046823] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6499.411298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6499.442501] alg: No test for adler32 (adler32-zlib) [ 6500.314211] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6500.403711] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6501.983135] Key type lgssc registered [ 6502.357271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.328193] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6519.095938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6519.100053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.170469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6520.179253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6520.211417] Lustre: lustre-MDT0000: new disk, initializing [ 6520.230454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.235667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6521.444595] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6526.916569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.011786] Lustre: lustre-OST0000: new disk, initializing [ 6527.013341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6527.014996] Lustre: Skipped 1 previous similar message [ 6527.016994] Lustre: 436417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6527.040802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.806637] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6532.362033] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6536.189421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6536.192598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6536.203546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6543.213932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.260233] Lustre: 435400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6543.289348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6543.331254] Lustre: lustre-MDT0001: new disk, initializing [ 6543.374868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6543.389913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6543.393099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6544.779240] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6548.523968] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6548.612310] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6550.060568] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6553.186382] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 6558.177933] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6558.178939] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.182816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6558.187052] Lustre: Skipped 1 previous similar message [ 6559.487834] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 6559.489885] Lustre: Skipped 1 previous similar message [ 6560.923619] Lustre: server umount lustre-MDT0001 complete [ 6568.417841] LustreError: 435404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.421971] LustreError: 435404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6573.536284] LustreError: 435410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.536635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.545042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.656676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.657308] LustreError: 435409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.342675] Lustre: server umount lustre-OST0000 complete [ 6586.803533] Lustre: server umount lustre-MDT0000 complete [ 6591.263039] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6592.322384] Key type lgssc unregistered [ 6592.455458] LNet: 440181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6592.458323] LNetError: 440181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6592.467301] LNet: Removed LNI 192.168.206.115@tcp [ 6592.781113] Key type .llcrypt unregistered [ 6592.782703] Key type ._llcrypt unregistered [ 6605.528201] Key type ._llcrypt registered [ 6605.529665] Key type .llcrypt registered [ 6605.571042] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6605.680524] Key type .llcrypt unregistered [ 6605.681609] Key type ._llcrypt unregistered [ 6612.601091] Key type ._llcrypt registered [ 6612.602200] Key type .llcrypt registered [ 6612.648222] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6613.155443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6613.168252] alg: No test for adler32 (adler32-zlib) [ 6614.035772] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6614.128468] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 6615.719127] Key type lgssc registered [ 6616.108347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6624.851568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6624.856188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.934773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6625.946616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6625.986312] Lustre: lustre-MDT0000: new disk, initializing [ 6626.009797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6626.017408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6627.386838] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6631.479588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.526011] Lustre: 443528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6631.549728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6631.552766] Lustre: Skipped 1 previous similar message [ 6631.588873] Lustre: lustre-MDT0001: new disk, initializing [ 6631.623993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6631.637451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6631.641672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6633.031170] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6636.744487] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6638.614610] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6640.815406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.913908] Lustre: lustre-OST0000: new disk, initializing [ 6640.915800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6640.919037] Lustre: 445498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6640.945909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.810622] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6643.960382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6643.964755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6643.996072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6646.540355] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.313150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.314871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6649.316598] Lustre: Skipped 2 previous similar messages [ 6649.319669] Lustre: Skipped 2 previous similar messages [ 6654.432254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.433723] Lustre: Skipped 1 previous similar message [ 6654.814106] Lustre: server umount lustre-MDT0000 complete [ 6656.154157] LustreError: 445497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316549 with bad export cookie 4202599548934160884 [ 6656.157559] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6656.159422] LustreError: 445497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6656.279244] Lustre: server umount lustre-MDT0001 complete [ 6667.731346] Lustre: server umount lustre-OST0000 complete [ 6671.042172] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:16:03 (1785316563) [ 6673.559196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.743969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.994995] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6677.553691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.941775] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6681.176286] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6682.745382] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6697.825861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.893290] Lustre: Found index 0 for lustre-OST0000, updating log [ 6697.952327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.954396] Lustre: Skipped 1 previous similar message [ 6699.902409] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6700.007264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6702.450257] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6705.119801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6705.123749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.130739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.711431] Lustre: server umount lustre-OST0000 complete [ 6722.237550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.238414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.242452] Lustre: Skipped 2 previous similar messages [ 6722.245738] Lustre: Skipped 3 previous similar messages [ 6727.648152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6727.650156] Lustre: Skipped 2 previous similar messages [ 6728.401593] Lustre: server umount lustre-MDT0000 complete [ 6729.650976] LustreError: 448255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316622 with bad export cookie 4202599548934162263 [ 6729.653909] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.654454] LustreError: 448255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6729.763923] Lustre: server umount lustre-MDT0001 complete [ 6732.600940] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:17:04 (1785316624) [ 6763.615268] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [ 7312.201527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.348299] LustreError: 453919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.354255] LustreError: 453919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7312.370368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.670937] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7316.368898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.866116] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7320.309055] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7322.194390] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.506721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.594173] Lustre: Found index 0 for lustre-OST0000, updating log [ 7337.696434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.698621] Lustre: Skipped 1 previous similar message [ 7339.653752] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7339.750865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7342.025305] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7343.778622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.783231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7348.817270] Lustre: server umount lustre-OST0000 complete [ 7350.240507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.244308] Lustre: Skipped 1 previous similar message [ 7356.182115] Lustre: server umount lustre-MDT0000 complete [ 7356.384307] LustreError: 455736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.389587] LustreError: 455736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7357.399414] LustreError: 455819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317250 with bad export cookie 4202599548934163327 [ 7357.403392] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7357.405140] LustreError: 455819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7357.524947] Lustre: server umount lustre-MDT0001 complete [ 7364.703240] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [ 7374.635456] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7377.853698] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7389.487211] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:28:01 (1785317281) [ 7395.161153] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7396.164373] Key type lgssc unregistered [ 7400.415358] LNet: 460572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7400.418106] LNetError: 460572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.029260] LNet: Removed LNI 192.168.206.115@tcp [ 7469.310555] Key type .llcrypt unregistered [ 7469.311399] Key type ._llcrypt unregistered [ 7475.592161] Key type ._llcrypt registered [ 7475.594106] Key type .llcrypt registered [ 7475.629457] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7475.891814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7475.901557] alg: No test for adler32 (adler32-zlib) [ 7476.741100] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7476.820810] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7478.391135] Key type lgssc registered [ 7478.723091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7480.027440] LNet: Added LNI 192.168.206.115@tcp100 [8/256/0/180] [ 7480.328150] LNet: Added LNI 192.168.206.115@tcp101 [8/256/0/180] [ 7480.611063] LNet: Added LNI 192.168.206.115@tcp102 [8/256/0/180] [ 7480.899655] LNet: Added LNI 192.168.206.115@tcp103 [8/256/0/180] [ 7481.177770] LNet: Added LNI 192.168.206.115@tcp104 [8/256/0/180] [ 7481.449112] LNet: Added LNI 192.168.206.115@tcp105 [8/256/0/180] [ 7481.715713] LNet: Added LNI 192.168.206.115@tcp106 [8/256/0/180] [ 7481.989464] LNet: Added LNI 192.168.206.115@tcp107 [8/256/0/180] [ 7482.261354] LNet: Added LNI 192.168.206.115@tcp108 [8/256/0/180] [ 7482.538400] LNet: Added LNI 192.168.206.115@tcp109 [8/256/0/180] [ 7482.821169] LNet: Added LNI 192.168.206.115@tcp110 [8/256/0/180] [ 7483.111917] LNet: Added LNI 192.168.206.115@tcp111 [8/256/0/180] [ 7483.405946] LNet: Added LNI 192.168.206.115@tcp112 [8/256/0/180] [ 7483.699484] LNet: Added LNI 192.168.206.115@tcp113 [8/256/0/180] [ 7483.998226] LNet: Added LNI 192.168.206.115@tcp114 [8/256/0/180] [ 7484.298804] LNet: Added LNI 192.168.206.115@tcp115 [8/256/0/180] [ 7484.598577] LNet: Added LNI 192.168.206.115@tcp116 [8/256/0/180] [ 7484.894636] LNet: Added LNI 192.168.206.115@tcp117 [8/256/0/180] [ 7485.188445] LNet: Added LNI 192.168.206.115@tcp118 [8/256/0/180] [ 7485.485658] LNet: Added LNI 192.168.206.115@tcp119 [8/256/0/180] [ 7485.778783] LNet: Added LNI 192.168.206.115@tcp120 [8/256/0/180] [ 7486.071817] LNet: Added LNI 192.168.206.115@tcp121 [8/256/0/180] [ 7486.372844] LNet: Added LNI 192.168.206.115@tcp122 [8/256/0/180] [ 7486.669536] LNet: Added LNI 192.168.206.115@tcp123 [8/256/0/180] [ 7486.964043] LNet: Added LNI 192.168.206.115@tcp124 [8/256/0/180] [ 7487.260576] LNet: Added LNI 192.168.206.115@tcp125 [8/256/0/180] [ 7487.553883] LNet: Added LNI 192.168.206.115@tcp126 [8/256/0/180] [ 7487.854568] LNet: Added LNI 192.168.206.115@tcp127 [8/256/0/180] [ 7488.159493] LNet: Added LNI 192.168.206.115@tcp128 [8/256/0/180] [ 7488.453347] LNet: Added LNI 192.168.206.115@tcp129 [8/256/0/180] [ 7488.756214] LNet: Added LNI 192.168.206.115@tcp130 [8/256/0/180] [ 7489.062521] LNet: Added LNI 192.168.206.115@tcp131 [8/256/0/180] [ 7489.364880] LNet: Added LNI 192.168.206.115@tcp132 [8/256/0/180] [ 7489.659563] LNet: Added LNI 192.168.206.115@tcp133 [8/256/0/180] [ 7489.956528] LNet: Added LNI 192.168.206.115@tcp134 [8/256/0/180] [ 7490.248965] LNet: Added LNI 192.168.206.115@tcp135 [8/256/0/180] [ 7492.462439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7492.468620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.507461] Lustre: 463567:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 7493.539022] LNetError: 463591:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7493.542145] LNetError: Protocol error connecting to 192.168.206.115@tcp115 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7493.545557] LNetError: Skipped 2 previous similar messages [ 7493.594248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7493.605743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7493.644712] Lustre: lustre-MDT0000: new disk, initializing [ 7493.667252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.672924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7494.239323] LNet: 7 local NIs in recovery (showing 7): 192.168.206.115@tcp122, 192.168.206.115@tcp125, 192.168.206.115@tcp126, 192.168.206.115@tcp127, 192.168.206.115@tcp121, 192.168.206.115@tcp124, 192.168.206.115@tcp129 [ 7494.241632] LNetError: 463598:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7494.250371] LNetError: 463598:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [ 7494.253113] LNetError: Protocol error connecting to 192.168.206.115@tcp114 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7494.256748] LNetError: Skipped 12 previous similar messages [ 7495.149560] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7495.264444] LNetError: 461032:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7495.268548] LNetError: 461032:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7495.271243] LNetError: Protocol error connecting to 192.168.206.115@tcp124 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7495.274594] LNetError: Skipped 2 previous similar messages [ 7499.344218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.394797] Lustre: 463582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7499.421746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7499.423550] Lustre: Skipped 1 previous similar message [ 7499.459383] Lustre: lustre-MDT0001: new disk, initializing [ 7499.470753] Lustre: 464388:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 7499.474194] Lustre: 464388:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7499.518501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.530834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7499.534014] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7500.385088] LNetError: 463591:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7500.389847] LNetError: 463591:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7500.393349] LNetError: Protocol error connecting to 192.168.206.115@tcp129 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7500.398685] LNetError: Skipped 1 previous similar message [ 7500.869326] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7502.258846] Lustre: 464754:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 7502.261974] Lustre: 464754:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7504.709119] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7506.628590] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7508.577221] LNetError: 461034:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7508.582178] LNetError: Protocol error connecting to 192.168.206.115@tcp129 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7508.915956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.045665] Lustre: lustre-OST0000: new disk, initializing [ 7509.048301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7509.051856] Lustre: 465561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7509.092428] Lustre: 465566:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [ 7509.095462] Lustre: 465566:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7509.114996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7509.764258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7509.768027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7509.821131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7511.151947] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7515.042654] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7520.225676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.227381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.228088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.228128] LustreError: Skipped 1 previous similar message [ 7520.231984] Lustre: Skipped 1 previous similar message [ 7523.286660] Lustre: server umount lustre-OST0000 complete [ 7523.937307] LNetError: 463594:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.115 [ 7523.941203] LNetError: Protocol error connecting to 192.168.206.115@tcp128 at host 192.168.206.115:988: is it running a compatible version of Lustre? [ 7525.345348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.345471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.346203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.346208] Lustre: Skipped 1 previous similar message [ 7525.350684] Lustre: Skipped 1 previous similar message [ 7530.464192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.466389] Lustre: Skipped 2 previous similar messages [ 7530.778402] Lustre: server umount lustre-MDT0000 complete [ 7531.994156] LustreError: 463576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317424 with bad export cookie 2106543316704985319 [ 7531.997804] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.998801] LustreError: 463576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7532.124623] Lustre: server umount lustre-MDT0001 complete [ 7535.704873] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7536.643678] Key type lgssc unregistered [ 7536.787198] LNet: 467472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.789660] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.791844] LNet: Removed LNI 192.168.206.115@tcp [ 7537.823311] LNet: Removed LNI 192.168.206.115@tcp100 [ 7537.825698] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7537.828092] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7537.830236] LNet: Removed LNI 192.168.206.115@tcp101 [ 7538.847177] LNet: Removed LNI 192.168.206.115@tcp102 [ 7538.848519] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.850527] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7538.852752] LNet: Removed LNI 192.168.206.115@tcp103 [ 7539.871139] LNet: Removed LNI 192.168.206.115@tcp104 [ 7540.895152] LNet: Removed LNI 192.168.206.115@tcp105 [ 7540.896463] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7540.898674] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7540.901029] LNet: Removed LNI 192.168.206.115@tcp106 [ 7541.919131] LNet: Removed LNI 192.168.206.115@tcp107 [ 7542.943198] LNet: Removed LNI 192.168.206.115@tcp108 [ 7543.967212] LNet: Removed LNI 192.168.206.115@tcp109 [ 7544.991149] LNet: Removed LNI 192.168.206.115@tcp110 [ 7544.993248] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7544.995537] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7544.997642] LNet: Removed LNI 192.168.206.115@tcp111 [ 7546.015113] LNet: Removed LNI 192.168.206.115@tcp112 [ 7547.039154] LNet: Removed LNI 192.168.206.115@tcp113 [ 7548.063209] LNet: Removed LNI 192.168.206.115@tcp114 [ 7549.087092] LNet: Removed LNI 192.168.206.115@tcp115 [ 7550.111127] LNet: Removed LNI 192.168.206.115@tcp116 [ 7551.135121] LNet: Removed LNI 192.168.206.115@tcp117 [ 7552.159193] LNet: Removed LNI 192.168.206.115@tcp118 [ 7553.183151] LNet: Removed LNI 192.168.206.115@tcp119 [ 7553.185054] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.188545] LNetError: 467472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7553.192381] LNet: Removed LNI 192.168.206.115@tcp120 [ 7554.207228] LNet: Removed LNI 192.168.206.115@tcp121 [ 7555.231185] LNet: Removed LNI 192.168.206.115@tcp122 [ 7556.255136] LNet: Removed LNI 192.168.206.115@tcp123 [ 7557.279178] LNet: Removed LNI 192.168.206.115@tcp124 [ 7558.303193] LNet: Removed LNI 192.168.206.115@tcp125 [ 7559.327106] LNet: Removed LNI 192.168.206.115@tcp126 [ 7560.351116] LNet: Removed LNI 192.168.206.115@tcp127 [ 7561.375091] LNet: Removed LNI 192.168.206.115@tcp128 [ 7562.399196] LNet: Removed LNI 192.168.206.115@tcp129 [ 7562.401158] LNet: Removed LNI 192.168.206.115@tcp130 [ 7562.403195] LNet: Removed LNI 192.168.206.115@tcp131 [ 7562.405358] LNet: Removed LNI 192.168.206.115@tcp132 [ 7562.406845] LNet: Removed LNI 192.168.206.115@tcp133 [ 7562.408225] LNet: Removed LNI 192.168.206.115@tcp134 [ 7562.415540] LNet: Removed LNI 192.168.206.115@tcp135 [ 7562.740768] Key type .llcrypt unregistered [ 7562.742211] Key type ._llcrypt unregistered [ 7569.887679] Key type ._llcrypt registered [ 7569.888783] Key type .llcrypt registered [ 7569.928680] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7575.963324] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7576.396854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7576.402866] alg: No test for adler32 (adler32-zlib) [ 7577.258766] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7577.353492] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7578.935110] Key type lgssc registered [ 7579.297198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.787108] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:31:22 (1785317482) [ 7597.060849] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7598.107075] Key type lgssc unregistered [ 7598.234314] LNet: 471531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7598.236603] LNetError: 471531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.243319] LNet: Removed LNI 192.168.206.115@tcp [ 7598.527441] Key type .llcrypt unregistered [ 7598.528472] Key type ._llcrypt unregistered [ 7605.203848] Key type ._llcrypt registered [ 7605.205660] Key type .llcrypt registered [ 7605.245088] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7605.600364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.636779] alg: No test for adler32 (adler32-zlib) [ 7606.497782] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7606.587037] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7608.167141] Key type lgssc registered [ 7608.549362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.640508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7611.645350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.729977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7612.740249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7612.778442] Lustre: lustre-MDT0000: new disk, initializing [ 7612.801177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.806849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7614.173136] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7618.220901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.260223] Lustre: 472718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7618.282678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7618.285906] Lustre: Skipped 1 previous similar message [ 7618.321852] Lustre: lustre-MDT0001: new disk, initializing [ 7618.352589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7618.364336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7618.369739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7619.691303] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7623.379416] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7625.214977] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7627.439909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.544567] Lustre: lustre-OST0000: new disk, initializing [ 7627.547684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7627.553192] Lustre: 474685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7627.583929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.554723] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7633.287533] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7634.495806] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lnet_if_list [ 7635.452103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7635.457196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7635.490407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7636.176546] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7636.187028] LNet: Added LNI 192.168.206.115@tcp10 [8/256/0/180] [ 7637.963287] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7637.973843] LNet: Added LNI 192.168.206.115@tcp11 [8/256/0/180] [ 7639.283474] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7639.296478] LNet: Added LNI 192.168.206.115@tcp12 [8/256/0/180] [ 7640.670086] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7640.682133] LNet: Added LNI 192.168.206.115@tcp13 [8/256/0/180] [ 7642.085321] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7642.099367] LNet: Added LNI 192.168.206.115@tcp14 [8/256/0/180] [ 7643.561406] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7643.572878] LNet: Added LNI 192.168.206.115@tcp15 [8/256/0/180] [ 7645.089637] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7645.100928] LNet: Added LNI 192.168.206.115@tcp16 [8/256/0/180] [ 7646.688599] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7646.702880] LNet: Added LNI 192.168.206.115@tcp17 [8/256/0/180] [ 7648.125290] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7648.134503] LNet: Added LNI 192.168.206.115@tcp18 [8/256/0/180] [ 7649.495570] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7649.505760] LNet: Added LNI 192.168.206.115@tcp19 [8/256/0/180] [ 7650.881157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7650.889672] LNet: Added LNI 192.168.206.115@tcp20 [8/256/0/180] [ 7652.482191] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7652.492265] LNet: Added LNI 192.168.206.115@tcp21 [8/256/0/180] [ 7654.165191] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7654.178360] LNet: Added LNI 192.168.206.115@tcp22 [8/256/0/180] [ 7655.988666] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7656.002428] LNet: Added LNI 192.168.206.115@tcp23 [8/256/0/180] [ 7657.989209] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7658.001706] LNet: Added LNI 192.168.206.115@tcp24 [8/256/0/180] [ 7659.889408] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7659.903525] LNet: Added LNI 192.168.206.115@tcp25 [8/256/0/180] [ 7661.870157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7661.884321] LNet: Added LNI 192.168.206.115@tcp26 [8/256/0/180] [ 7663.565910] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7663.575997] LNet: Added LNI 192.168.206.115@tcp27 [8/256/0/180] [ 7665.217994] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7665.227100] LNet: Added LNI 192.168.206.115@tcp28 [8/256/0/180] [ 7666.943717] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7666.954979] LNet: Added LNI 192.168.206.115@tcp29 [8/256/0/180] [ 7668.662530] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7668.671860] LNet: Added LNI 192.168.206.115@tcp30 [8/256/0/180] [ 7670.458804] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7670.468899] LNet: Added LNI 192.168.206.115@tcp31 [8/256/0/180] [ 7672.225624] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7672.234812] LNet: Added LNI 192.168.206.115@tcp32 [8/256/0/180] [ 7674.033275] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7674.041530] LNet: Added LNI 192.168.206.115@tcp33 [8/256/0/180] [ 7676.107408] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7676.118694] LNet: Added LNI 192.168.206.115@tcp34 [8/256/0/180] [ 7678.296815] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7678.309794] LNet: Added LNI 192.168.206.115@tcp35 [8/256/0/180] [ 7680.598435] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7680.611992] LNet: Added LNI 192.168.206.115@tcp36 [8/256/0/180] [ 7682.980774] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7682.993465] LNet: Added LNI 192.168.206.115@tcp37 [8/256/0/180] [ 7685.313494] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7685.324028] LNet: Added LNI 192.168.206.115@tcp38 [8/256/0/180] [ 7687.712344] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7687.723036] LNet: Added LNI 192.168.206.115@tcp39 [8/256/0/180] [ 7690.050547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7690.061599] LNet: Added LNI 192.168.206.115@tcp40 [8/256/0/180] [ 7692.677671] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7692.690747] LNet: Added LNI 192.168.206.115@tcp41 [8/256/0/180] [ 7695.119020] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7695.130863] LNet: Added LNI 192.168.206.115@tcp42 [8/256/0/180] [ 7697.659818] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7697.671848] LNet: Added LNI 192.168.206.115@tcp43 [8/256/0/180] [ 7700.062550] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7700.074799] LNet: Added LNI 192.168.206.115@tcp44 [8/256/0/180] [ 7702.604418] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7702.615865] LNet: Added LNI 192.168.206.115@tcp45 [8/256/0/180] [ 7705.116401] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7705.128861] LNet: Added LNI 192.168.206.115@tcp46 [8/256/0/180] [ 7707.752266] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7707.764369] LNet: Added LNI 192.168.206.115@tcp47 [8/256/0/180] [ 7710.630809] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7710.648528] LNet: Added LNI 192.168.206.115@tcp48 [8/256/0/180] [ 7713.529370] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7713.541093] LNet: Added LNI 192.168.206.115@tcp49 [8/256/0/180] [ 7715.991536] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7716.003407] LNet: Added LNI 192.168.206.115@tcp50 [8/256/0/180] [ 7726.047847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7726.050883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.056493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.584123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.585404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7729.105721] Lustre: server umount lustre-MDT0000 complete [ 7730.253167] LustreError: 472713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317623 with bad export cookie 4117959348443856735 [ 7730.255738] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.256149] LustreError: 472713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7730.361109] Lustre: server umount lustre-MDT0001 complete [ 7737.691641] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7740.908857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.023108] Lustre: 471994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317625/real 1785317625] req@0000000026376d07 x1872041081911936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7749.031095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.034737] Lustre: Skipped 1 previous similar message [ 7752.095099] Lustre: 471994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317628/real 1785317628] req@00000000dfa70104 x1872041081912064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7753.183168] Lustre: 471995:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317630/real 1785317630] req@000000009f934676 x1872041081912192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7755.167512] LustreError: 471992:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009b9e0a90 x1872041081913088/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7755.279318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.255116] Lustre: 471996:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317633/real 1785317633] req@000000005e33b19a x1872041081912448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7756.618738] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7759.507349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.656841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7761.134319] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7761.376123] Lustre: 471993:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317638/real 1785317638] req@00000000e87a7576 x1872041081912576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7763.677365] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7764.967106] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7765.564313] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7768.036895] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7775.201967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7775.204302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7775.208609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7775.210419] Lustre: Skipped 2 previous similar messages [ 7778.274260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7778.276260] Lustre: Skipped 2 previous similar messages [ 7779.418323] Lustre: server umount lustre-OST0000 complete [ 7783.393622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.394152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.397349] Lustre: Skipped 2 previous similar messages [ 7786.836988] Lustre: server umount lustre-MDT0000 complete [ 7788.119467] LustreError: 473514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317681 with bad export cookie 4117959348443858723 [ 7788.123238] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7788.123638] LustreError: 473514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7788.257673] Lustre: server umount lustre-MDT0001 complete [ 7792.129458] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7793.236517] Key type lgssc unregistered [ 7793.413410] LNet: 501644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7793.418034] LNetError: 501644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7793.420979] LNet: Removed LNI 192.168.206.115@tcp [ 7793.422085] LNet: Removed LNI 192.168.206.115@tcp10 [ 7793.423148] LNet: Removed LNI 192.168.206.115@tcp11 [ 7793.424237] LNet: Removed LNI 192.168.206.115@tcp12 [ 7793.425290] LNet: Removed LNI 192.168.206.115@tcp13 [ 7793.426307] LNet: Removed LNI 192.168.206.115@tcp14 [ 7793.427405] LNet: Removed LNI 192.168.206.115@tcp15 [ 7793.428438] LNet: Removed LNI 192.168.206.115@tcp16 [ 7793.429412] LNet: Removed LNI 192.168.206.115@tcp17 [ 7793.430428] LNet: Removed LNI 192.168.206.115@tcp18 [ 7793.431495] LNet: Removed LNI 192.168.206.115@tcp19 [ 7793.432603] LNet: Removed LNI 192.168.206.115@tcp20 [ 7793.433666] LNet: Removed LNI 192.168.206.115@tcp21 [ 7793.434759] LNet: Removed LNI 192.168.206.115@tcp22 [ 7793.435854] LNet: Removed LNI 192.168.206.115@tcp23 [ 7793.436881] LNet: Removed LNI 192.168.206.115@tcp24 [ 7793.437937] LNet: Removed LNI 192.168.206.115@tcp25 [ 7793.439042] LNet: Removed LNI 192.168.206.115@tcp26 [ 7793.440149] LNet: Removed LNI 192.168.206.115@tcp27 [ 7793.442356] LNet: Removed LNI 192.168.206.115@tcp28 [ 7793.444568] LNet: Removed LNI 192.168.206.115@tcp29 [ 7793.446824] LNet: Removed LNI 192.168.206.115@tcp30 [ 7793.449071] LNet: Removed LNI 192.168.206.115@tcp31 [ 7793.450272] LNet: Removed LNI 192.168.206.115@tcp32 [ 7793.451461] LNet: Removed LNI 192.168.206.115@tcp33 [ 7793.452704] LNet: Removed LNI 192.168.206.115@tcp34 [ 7793.454140] LNet: Removed LNI 192.168.206.115@tcp35 [ 7793.456229] LNet: Removed LNI 192.168.206.115@tcp36 [ 7793.458396] LNet: Removed LNI 192.168.206.115@tcp37 [ 7793.459506] LNet: Removed LNI 192.168.206.115@tcp38 [ 7793.460843] LNet: Removed LNI 192.168.206.115@tcp39 [ 7793.461967] LNet: Removed LNI 192.168.206.115@tcp40 [ 7793.463764] LNet: Removed LNI 192.168.206.115@tcp41 [ 7793.466111] LNet: Removed LNI 192.168.206.115@tcp42 [ 7793.469557] LNet: Removed LNI 192.168.206.115@tcp43 [ 7793.471456] LNet: Removed LNI 192.168.206.115@tcp44 [ 7793.473406] LNet: Removed LNI 192.168.206.115@tcp45 [ 7793.475304] LNet: Removed LNI 192.168.206.115@tcp46 [ 7793.476586] LNet: Removed LNI 192.168.206.115@tcp47 [ 7793.477822] LNet: Removed LNI 192.168.206.115@tcp48 [ 7793.479216] LNet: Removed LNI 192.168.206.115@tcp49 [ 7793.485296] LNet: Removed LNI 192.168.206.115@tcp50 [ 7793.763851] Key type .llcrypt unregistered [ 7793.764798] Key type ._llcrypt unregistered [ 7800.376091] Key type ._llcrypt registered [ 7800.377777] Key type .llcrypt registered [ 7800.420339] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7806.804895] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7807.268398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7807.281962] alg: No test for adler32 (adler32-zlib) [ 7808.137450] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7808.225982] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7809.807145] Key type lgssc registered [ 7810.209600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7821.621137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7821.625747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.702822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7822.711847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7822.747075] Lustre: lustre-MDT0000: new disk, initializing [ 7822.765456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.770789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7824.031651] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7828.193029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.237232] Lustre: 504815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7828.265721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7828.269028] Lustre: Skipped 1 previous similar message [ 7828.312475] Lustre: lustre-MDT0001: new disk, initializing [ 7828.350338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7828.365500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7828.369355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7829.821309] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7833.620856] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7835.468385] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7838.172024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7838.254185] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7838.284689] Lustre: lustre-OST0000: new disk, initializing [ 7838.286523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7838.289784] Lustre: 506828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7838.321066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7840.242538] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7840.695214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7840.699870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7840.738512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7844.024908] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.978867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.979239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7850.987404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7850.988974] LustreError: Skipped 1 previous similar message [ 7850.990966] Lustre: Skipped 1 previous similar message [ 7852.500113] Lustre: server umount lustre-OST0000 complete [ 7854.048090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7854.050501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.056541] Lustre: Skipped 1 previous similar message [ 7854.059210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7856.096152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7856.097091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.216126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.218109] Lustre: Skipped 2 previous similar messages [ 7866.336551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7866.338741] Lustre: Skipped 1 previous similar message [ 7868.383233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7868.455078] Lustre: server umount lustre-MDT0000 complete [ 7869.761041] LustreError: 504807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317762 with bad export cookie 12322269337652302976 [ 7869.763533] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7869.766174] LustreError: 504807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7869.897938] Lustre: server umount lustre-MDT0001 complete [ 7873.562693] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7874.620274] Key type lgssc unregistered [ 7874.754361] LNet: 508733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7874.756529] LNetError: 508733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7874.764378] LNet: Removed LNI 192.168.206.115@tcp [ 7875.075787] Key type .llcrypt unregistered [ 7875.076723] Key type ._llcrypt unregistered [ 7882.666743] Key type ._llcrypt registered [ 7882.667886] Key type .llcrypt registered [ 7882.709438] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7883.143163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7883.216676] alg: No test for adler32 (adler32-zlib) [ 7884.075079] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7884.166748] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7885.743201] Key type lgssc registered [ 7886.109799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7888.758891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7888.766081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.936160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.369899] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7894.214129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.348897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7895.751256] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7898.232171] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7900.097482] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7902.250381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.365484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.403798] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7906.865622] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7920.992044] loop0: detected capacity change from 0 to 204800000 [ 7921.112192] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7921.115561] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7921.119189] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7923.130790] loop0: detected capacity change from 0 to 204800000 [ 7923.195263] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7923.200511] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7923.205167] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7925.106445] loop0: detected capacity change from 0 to 204800000 [ 7927.171609] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.221529] Lustre: fs15246-OST0000: new disk, initializing [ 7927.223244] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7927.226233] Lustre: 513593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7927.246782] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.284555] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7933.205754] loop1: detected capacity change from 0 to 204800000 [ 7935.114214] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.164229] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7935.188157] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7935.230900] Lustre: fs15246-MDT0000: new disk, initializing [ 7935.289364] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.301666] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7935.358496] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7936.416937] Lustre: 514714:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000508ed82b x1872041373366144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785317845 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7936.748514] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7941.472527] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7941.475109] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7941.482746] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7946.212511] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.218599] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.844361] Lustre: server umount fs15246-MDT0000 complete [ 7951.332513] LustreError: 509974:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.338464] LustreError: 509974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7953.044819] Lustre: server umount fs15246-OST0000 complete [ 7961.056868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7961.057918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.059712] LustreError: Skipped 1 previous similar message [ 7961.065611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.069085] Lustre: Skipped 1 previous similar message [ 7962.704422] Lustre: server umount lustre-OST0000 complete [ 7966.175932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7966.179321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.183323] Lustre: Skipped 1 previous similar message [ 7966.185169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.202274] Lustre: server umount lustre-MDT0000 complete [ 7971.491150] LustreError: 509961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317864 with bad export cookie 5487441254954006426 [ 7971.494468] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7971.496272] LustreError: 509961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7971.608940] Lustre: server umount lustre-MDT0001 complete [ 7974.847962] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7975.766188] Key type lgssc unregistered [ 7975.878350] LNet: 517805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7975.881130] LNetError: 517805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7975.890274] LNet: Removed LNI 192.168.206.115@tcp [ 7976.190917] Key type .llcrypt unregistered [ 7976.192358] Key type ._llcrypt unregistered [ 7982.939640] Key type ._llcrypt registered [ 7982.940822] Key type .llcrypt registered [ 7982.982030] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7983.308961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7983.374586] alg: No test for adler32 (adler32-zlib) [ 7984.220386] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7984.304354] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7985.879142] Key type lgssc registered [ 7986.212371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7997.594396] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8000.555872] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8011.448059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8011.452071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.523239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8012.532567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8012.563580] Lustre: lustre-MDT0000: new disk, initializing [ 8012.582817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.590244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8013.795635] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8017.909750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.951732] Lustre: 522239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8017.978563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8017.980570] Lustre: Skipped 1 previous similar message [ 8018.022756] Lustre: lustre-MDT0001: new disk, initializing [ 8018.064523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8018.080879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8018.084314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8019.482170] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8023.233192] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8025.087961] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8027.370601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.471174] Lustre: lustre-OST0000: new disk, initializing [ 8027.473679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8027.476113] Lustre: 524207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8027.501631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.316282] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8032.925361] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.474912] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8035.318472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8035.323117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8035.350770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8035.569980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8036.983465] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8037.065980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8040.418069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8040.419194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.420974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8040.425116] Lustre: Skipped 1 previous similar message [ 8043.801258] Lustre: server umount lustre-OST0000 complete [ 8045.536474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.537285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.540103] Lustre: Skipped 1 previous similar message [ 8045.545700] Lustre: Skipped 2 previous similar messages [ 8050.656053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.658133] Lustre: Skipped 1 previous similar message [ 8051.283027] Lustre: server umount lustre-MDT0000 complete [ 8052.522300] LustreError: 522277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317945 with bad export cookie 12241923026555744261 [ 8052.525131] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.525885] LustreError: 522277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8052.649259] Lustre: server umount lustre-MDT0001 complete [ 8060.353477] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8063.254404] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8073.694906] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7814 sec ========== 05:39:25 (1785317965) [ 8074.262674] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:39:26 (1785317966) === [ 8075.434338] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:39:27 (1785317967) === [ 8088.170133] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8089.109702] Key type lgssc unregistered [ 8089.236299] LNet: 530846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8089.238599] LNetError: 530846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8089.247288] LNet: Removed LNI 192.168.206.115@tcp [ 8089.541562] Key type .llcrypt unregistered [ 8089.542931] Key type ._llcrypt unregistered