[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487016510 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003193] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014060] *** VALIDATE selinux *** [ 0.022736] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026193] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] *** VALIDATE tmpfs *** [ 0.031289] *** VALIDATE proc *** [ 0.032161] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.041799] debug: unmapping init [mem 0xffffffffac259000-0xffffffffac260fff] [ 0.043155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044608] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049007] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052290] rcu: Hierarchical SRCU implementation. [ 0.054469] smp: Bringing up secondary CPUs ... [ 0.055588] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059215] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147508] node 0 deferred pages initialised in 81ms [ 0.151012] devtmpfs: initialized [ 0.152262] x86/mm: Memory block size: 128MB [ 0.154885] gcov: version magic: 0x41383552 [ 0.156266] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158251] pinctrl core: initialized pinctrl subsystem [ 0.159425] [ 0.160010] ************************************************************* [ 0.161014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162014] ** ** [ 0.163012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164018] ** ** [ 0.165014] ** This means that this kernel is built to expose internal ** [ 0.166013] ** IOMMU data structures, which may compromise security on ** [ 0.167017] ** your system. ** [ 0.168014] ** ** [ 0.169015] ** If you see this message and you are not debugging the ** [ 0.170014] ** kernel, report this immediately to your vendor! ** [ 0.171015] ** ** [ 0.172014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173013] ************************************************************* [ 0.174797] NET: Registered protocol family 16 [ 0.175476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178584] cpuidle: using governor menu [ 0.180935] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184436] PCI: Using configuration type 1 for base access [ 0.186119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199044] cryptd: max_cpu_qlen set to 1000 [ 0.202151] ACPI: Added _OSI(Module Device) [ 0.205018] ACPI: Added _OSI(Processor Device) [ 0.206013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213767] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220354] ACPI: Interpreter enabled [ 0.221065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223018] ACPI: Using IOAPIC for interrupt routing [ 0.225100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240511] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.256674] acpiphp: Slot [2] registered [ 0.258270] acpiphp: Slot [5] registered [ 0.260173] acpiphp: Slot [6] registered [ 0.262185] acpiphp: Slot [7] registered [ 0.263178] acpiphp: Slot [8] registered [ 0.265183] acpiphp: Slot [9] registered [ 0.267225] acpiphp: Slot [10] registered [ 0.269199] acpiphp: Slot [3] registered [ 0.270200] acpiphp: Slot [4] registered [ 0.272168] acpiphp: Slot [11] registered [ 0.274124] acpiphp: Slot [12] registered [ 0.276101] acpiphp: Slot [13] registered [ 0.277082] acpiphp: Slot [14] registered [ 0.278131] acpiphp: Slot [15] registered [ 0.279086] acpiphp: Slot [16] registered [ 0.281083] acpiphp: Slot [17] registered [ 0.282075] acpiphp: Slot [18] registered [ 0.283161] acpiphp: Slot [19] registered [ 0.285178] acpiphp: Slot [20] registered [ 0.287204] acpiphp: Slot [21] registered [ 0.289225] acpiphp: Slot [22] registered [ 0.291347] acpiphp: Slot [23] registered [ 0.293102] acpiphp: Slot [24] registered [ 0.294097] acpiphp: Slot [25] registered [ 0.296087] acpiphp: Slot [26] registered [ 0.297102] acpiphp: Slot [27] registered [ 0.300156] acpiphp: Slot [28] registered [ 0.301124] acpiphp: Slot [29] registered [ 0.303126] acpiphp: Slot [30] registered [ 0.305115] acpiphp: Slot [31] registered [ 0.307105] PCI host bridge to bus 0000:00 [ 0.308040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327199] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330450] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340799] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345943] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.360534] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364014] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369732] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388914] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402488] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414094] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454311] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491068] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504526] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.557455] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.563014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.566998] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.592833] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.597933] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.603014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.619016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.628022] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.650024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686303] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709024] iommu: Default domain type: Passthrough [ 0.710567] SCSI subsystem initialized [ 0.712151] ACPI: bus type USB registered [ 0.714138] usbcore: registered new interface driver usbfs [ 0.716082] usbcore: registered new interface driver hub [ 0.719094] usbcore: registered new device driver usb [ 0.721192] pps_core: LinuxPPS API ver. 1 registered [ 0.723012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.726060] PTP clock support registered [ 0.729035] EDAC MC: Ver: 3.0.0 [ 0.731160] PCI: Using ACPI for IRQ routing [ 0.733684] NetLabel: Initializing [ 0.734010] NetLabel: domain hash size = 128 [ 0.736025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738130] NetLabel: unlabeled traffic allowed by default [ 0.740155] vgaarb: loaded [ 0.742008] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.744013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.752205] clocksource: Switched to clocksource kvm-clock [ 0.854091] VFS: Disk quotas dquot_6.6.0 [ 0.855570] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857968] *** VALIDATE ramfs *** [ 0.861643] *** VALIDATE hugetlbfs *** [ 0.863499] pnp: PnP ACPI init [ 0.866676] pnp: PnP ACPI: found 6 devices [ 0.884262] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.888120] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.890694] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.893346] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.896027] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.899524] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.902628] NET: Registered protocol family 2 [ 0.905353] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.910558] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.914523] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.920397] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.924377] TCP: Hash tables configured (established 65536 bind 65536) [ 0.927570] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.931121] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934294] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.937626] NET: Registered protocol family 1 [ 0.941261] RPC: Registered named UNIX socket transport module. [ 0.943780] RPC: Registered udp transport module. [ 0.945674] RPC: Registered tcp transport module. [ 0.947627] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.950174] NET: Registered protocol family 44 [ 0.951983] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.953960] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.956330] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.958871] PCI: CLS 0 bytes, default 64 [ 0.960559] Unpacking initramfs... [ 2.383822] debug: unmapping init [mem 0xffff9d8dfcc54000-0xffff9d8dfffbffff] [ 2.389182] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.391824] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.394334] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.916603] Initialise system trusted keyrings [ 2.918694] Key type blacklist registered [ 2.920943] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.930705] zbud: loaded [ 2.935347] *** VALIDATE nfs *** [ 2.936914] *** VALIDATE nfs4 *** [ 2.938863] pstore: using deflate compression [ 2.942502] Platform Keyring initialized [ 3.058867] NET: Registered protocol family 38 [ 3.061282] Key type asymmetric registered [ 3.063334] Asymmetric key parser 'x509' registered [ 3.066546] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.071454] io scheduler mq-deadline registered [ 3.073813] io scheduler kyber registered [ 3.076189] io scheduler bfq registered [ 3.078613] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.082469] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.085193] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.088380] ACPI: Power Button [PWRF] [ 3.094265] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.100779] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.113040] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.119753] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.132462] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.159733] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.188480] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.194346] Non-volatile memory driver v1.3 [ 3.196498] Linux agpgart interface v0.103 [ 3.228797] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.232012] vda: detected capacity change from 0 to 76677120 [ 3.245595] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.249323] vdb: detected capacity change from 0 to 1073741824 [ 3.267945] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.270747] vdc: detected capacity change from 0 to 2621440000 [ 3.285385] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.288621] vdd: detected capacity change from 0 to 2621440000 [ 3.304401] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.307219] vde: detected capacity change from 0 to 4294967296 [ 3.322689] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.326032] vdf: detected capacity change from 0 to 4294967296 [ 3.331411] libphy: Fixed MDIO Bus: probed [ 3.340308] usbcore: registered new interface driver usbserial_generic [ 3.342695] usbserial: USB Serial support registered for generic [ 3.344544] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348214] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.349533] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.351727] mousedev: PS/2 mouse device common for all mice [ 3.354925] rtc_cmos 00:05: RTC can wake from S4 [ 3.357814] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.362073] rtc_cmos 00:05: registered as rtc0 [ 3.364316] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.365328] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.368842] intel_pstate: CPU model not supported [ 3.374107] hid: raw HID events driver (C) Jiri Kosina [ 3.374331] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.375791] usbcore: registered new interface driver usbhid [ 3.379561] usbhid: USB HID core driver [ 3.381275] drop_monitor: Initializing network drop monitor service [ 3.383579] Initializing XFRM netlink socket [ 3.385120] NET: Registered protocol family 10 [ 3.387826] Segment Routing with IPv6 [ 3.389280] NET: Registered protocol family 17 [ 3.391316] mpls_gso: MPLS GSO support [ 3.395898] RAS: Correctable Errors collector initialized. [ 3.398483] AVX version of gcm_enc/dec engaged. [ 3.400493] AES CTR mode by8 optimization enabled [ 3.478914] sched_clock: Marking stable (3478882720, 0)->(4387198820, -908316100) [ 3.482123] registered taskstats version 1 [ 3.484136] Loading compiled-in X.509 certificates [ 3.486628] zswap: loaded using pool lzo/zbud [ 3.509706] Key type big_key registered [ 3.521315] Key type encrypted registered [ 3.523056] ima: No TPM chip found, activating TPM-bypass! [ 3.525417] ima: Allocated hash algorithm: sha1 [ 3.527232] ima: No architecture policies found [ 3.529225] evm: Initialising EVM extended attributes: [ 3.531252] evm: security.selinux [ 3.532609] evm: security.ima [ 3.533783] evm: security.capability [ 3.535339] evm: HMAC attrs: 0x1 [ 3.538152] rtc_cmos 00:05: setting system clock to 2026-09-04 16:47:57 UTC (1788540477) [ 3.544846] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 3.548490] debug: unmapping init [mem 0xffffffffabf82000-0xffffffffac258fff] [ 3.557407] Write protecting the kernel read-only data: 28672k [ 3.561033] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 3.564042] debug: unmapping init [mem 0xffffffffaaf14000-0xffffffffaaffffff] [ 3.604787] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.613984] systemd[1]: Detected virtualization kvm. [ 3.616112] systemd[1]: Detected architecture x86-64. [ 3.617632] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.645619] systemd[1]: No hostname configured. [ 3.647487] systemd[1]: Set hostname to . [ 3.649777] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651876] systemd[1]: Initializing machine ID from random generator. [ 3.697900] random: ln: uninitialized urandom read (6 bytes read) [ 3.783290] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787490] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.795669] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.801600] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.473295] device-mapper: uevent: version 1.0.3 [ 4.476125] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.140335] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.209339] virtio_net virtio0 ens2: renamed from eth0 [ 5.229166] scsi host0: ata_piix [ 5.246453] scsi host1: ata_piix [ 5.249818] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.254284] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.530739] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.039927] random: crng init done [ 10.041530] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.408510] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.536386] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.795531] SELinux: Disabled at runtime. [ 11.863111] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.871212] systemd[1]: Detected virtualization kvm. [ 11.872515] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.372602] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.376396] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.382536] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.387408] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.392413] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.399244] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.409150] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ 12.538736] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.868414] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.252271] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.272110] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.431982] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.444664] EDAC sbridge: Ver: 1.1.2 [ 15.407397] Key type dns_resolver registered [ 15.713457] NFS: Registering the id_resolver key type [ 15.715968] Key type id_resolver registered [ 15.717861] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg358-server login: [ 49.450575] libcfs: loading out-of-tree module taints kernel. [ 49.493412] Key type ._llcrypt registered [ 49.495532] Key type .llcrypt registered [ 49.571471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 66.430927] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 67.661332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.671408] alg: No test for adler32 (adler32-zlib) [ 68.077015] hrtimer: interrupt took 12998111 ns [ 69.139691] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 69.852800] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 71.632143] Key type lgssc registered [ 72.982861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.749514] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.850805] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 138.128200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.166230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.374365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.401214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.465449] Lustre: lustre-MDT0000: new disk, initializing [ 139.531708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.548750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.533662] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.868841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.972825] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.006581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.009506] Lustre: Skipped 1 previous similar message [ 158.127676] Lustre: lustre-MDT0001: new disk, initializing [ 158.238864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.278694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.307617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.328792] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.739251] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.532701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.013904] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 177.023808] Lustre: lustre-OST0000: new disk, initializing [ 177.039645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.047134] Lustre: 8441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.143627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.273717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.281876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.348545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.336325] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.454082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.630731] Lustre: lustre-OST0001: new disk, initializing [ 197.636374] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.649127] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.752368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.429646] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.848816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.860469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.894303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.119661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.797280] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 230.202408] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.533771] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing yml_node [ 240.689352] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 243.285539] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 246.370180] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 248.683523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 12:52:00 EDT 2026 [ 266.792758] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 268.561323] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.668912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.672059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.687759] Lustre: Skipped 2 previous similar messages [ 285.703891] Lustre: Skipped 3 previous similar messages [ 290.583914] Lustre: server umount lustre-MDT0000 complete [ 293.856171] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540767 with bad export cookie 14228677776764414305 [ 293.858313] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.863479] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 295.904979] LustreError: 8012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.909155] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.911929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.936780] LustreError: 8012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 295.972706] Lustre: Skipped 2 previous similar messages [ 298.913527] LustreError: 6511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.913598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.945298] Lustre: Skipped 2 previous similar messages [ 304.032900] LustreError: 6509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.034139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.052519] LustreError: 6509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 304.066617] Lustre: Skipped 1 previous similar message [ 308.192300] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 308.418589] Lustre: server umount lustre-MDT0001 complete [ 311.982533] Lustre: server umount lustre-OST0000 complete [ 315.199988] Lustre: server umount lustre-OST0001 complete [ 320.527303] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 327.988025] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 360.866545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.086283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 361.116685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 361.178349] Lustre: lustre-MDT0000: new disk, initializing [ 361.235388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.249215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 365.078547] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 373.459802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.625744] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.694849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.700268] Lustre: Skipped 1 previous similar message [ 373.783788] Lustre: lustre-MDT0001: new disk, initializing [ 373.864264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 373.905791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 373.922085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.921183] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 384.424534] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 389.356349] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 395.258286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 395.434303] Lustre: lustre-OST0000: new disk, initializing [ 395.437899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 395.443215] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 395.518117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.281819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 397.291365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 397.382857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.003496] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 408.164158] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 412.224506] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 412.499186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 415.918460] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 416.113321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 417.763173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 417.770498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.783616] LustreError: Skipped 1 previous similar message [ 417.789765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 417.800947] Lustre: Skipped 1 previous similar message [ 423.703484] Lustre: server umount lustre-OST0000 complete [ 428.003571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.242121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 438.244948] Lustre: Skipped 7 previous similar messages [ 442.336281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.435387] Lustre: server umount lustre-MDT0000 complete [ 443.368810] LustreError: 15074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.395996] LustreError: 15074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 446.406768] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788540920 with bad export cookie 14228677776764415838 [ 446.411676] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.422537] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 446.816789] Lustre: server umount lustre-MDT0001 complete [ 451.120337] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:55:23 (1788540923) [ 453.920295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 456.329884] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:55:28 (1788540928) [ 463.325986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.759878] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.862446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.895768] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 473.058352] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.080160] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 477.135207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.759964] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 488.649344] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 493.806262] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.855389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 507.327264] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 515.462775] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.752692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.758775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.769628] Lustre: Skipped 2 previous similar messages [ 523.770735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.794681] Lustre: Skipped 1 previous similar message [ 529.573151] Lustre: server umount lustre-OST0000 complete [ 533.988974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.012281] Lustre: Skipped 1 previous similar message [ 534.018601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.544514] Lustre: server umount lustre-MDT0000 complete [ 543.526366] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541017 with bad export cookie 14228677776764416720 [ 543.539709] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 543.541560] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.947825] Lustre: server umount lustre-MDT0001 complete [ 554.459688] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 557.337337] Key type lgssc unregistered [ 557.622206] LNet: 22979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.631157] LNetError: 22979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.643586] LNet: Removed LNI 192.168.203.158@tcp [ 558.648159] Key type .llcrypt unregistered [ 558.650557] Key type ._llcrypt unregistered [ 578.763519] Key type ._llcrypt registered [ 578.766953] Key type .llcrypt registered [ 578.857396] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 580.081088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 580.107716] alg: No test for adler32 (adler32-zlib) [ 581.130953] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 581.353959] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 583.024151] Key type lgssc registered [ 583.893586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 591.008152] loop: module loaded [ 591.094352] loop0: detected capacity change from 0 to 204800000 [ 591.334975] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.347402] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.678326] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 596.840828] loop0: detected capacity change from 0 to 204800000 [ 597.031466] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 597.050543] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 597.071508] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 601.863700] loop0: detected capacity change from 0 to 204800000 [ 606.847526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.886303] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.173696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 608.217424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 608.300507] Lustre: lustre-MDT0000: new disk, initializing [ 608.368747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.386526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 612.714582] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 620.926879] loop1: detected capacity change from 0 to 204800000 [ 626.565354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.112421] Lustre: lustre-OST07c6: new disk, initializing [ 627.118871] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 627.129466] Lustre: Skipped 1 previous similar message [ 627.140910] Lustre: 26140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 627.256092] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 634.157179] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 635.988184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 636.001021] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 636.037219] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 639.592626] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 639.605316] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 639.611572] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 639.616156] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 639.627675] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 639.638284] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 654.874595] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 654.885168] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 654.894380] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 654.907667] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 654.913689] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 654.919567] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.804062] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 659.823669] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 659.828280] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 659.834273] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.842964] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.848705] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.858292] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 659.869959] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.881871] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 659.890937] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.902101] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.911794] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.120725] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 663.135542] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 663.143334] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 663.151346] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.157220] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.167123] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.178780] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 663.187520] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.195680] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 663.209090] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.218328] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 663.224237] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.929636] Lustre: 26586:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 667.938123] Lustre: 26586:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 667.943325] Lustre: 26586:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 667.951783] Lustre: 26586:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.957955] Lustre: 26586:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 667.962777] Lustre: 26586:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.971111] Lustre: 26586:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 667.978324] Lustre: 26586:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.983989] Lustre: 26586:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 667.989877] Lustre: 26586:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.996568] Lustre: 26586:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.005487] Lustre: 26586:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.409984] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 679.424566] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 679.436827] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.441897] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.446860] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.452219] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.457414] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 679.463504] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.470903] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 679.481442] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.491590] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.502188] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.033165] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 728.041115] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.101257] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 733.155642] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 738.276119] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 741.344349] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 741.657458] Lustre: server umount lustre-OST07c6 complete [ 751.797618] Lustre: server umount lustre-MDT0000 complete [ 768.158675] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 770.870808] Key type lgssc unregistered [ 771.151196] LNet: 28704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.160781] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.202355] LNet: Removed LNI 192.168.203.158@tcp [ 773.116633] Key type .llcrypt unregistered [ 773.119059] Key type ._llcrypt unregistered [ 792.491058] Key type ._llcrypt registered [ 792.494543] Key type .llcrypt registered [ 792.616888] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 794.107253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.124550] alg: No test for adler32 (adler32-zlib) [ 795.182218] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 795.525510] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 797.184172] Key type lgssc registered [ 798.578966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.596212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 806.615582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.042600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.109490] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 817.636823] LustreError: 29985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.662575] LustreError: 29985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 820.526284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.919593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 825.862796] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 832.818213] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 838.204676] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.368503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.715091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.328481] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 852.935044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 859.266541] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 877.543046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.560431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.581583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.566631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.576501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.597494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.267923] Lustre: server umount lustre-OST0000 complete [ 887.266946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 887.275128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.295494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.302470] Lustre: Skipped 1 previous similar message [ 890.726220] Lustre: server umount lustre-MDT0000 complete [ 893.931168] LustreError: 29982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.970175] LustreError: 29982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 895.081482] LustreError: 29964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541369 with bad export cookie 14778525491509142161 [ 895.083554] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.091022] LustreError: 29964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.537950] Lustre: server umount lustre-MDT0001 complete [ 907.407614] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 910.297599] Key type lgssc unregistered [ 910.625653] LNet: 33720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.633580] LNetError: 33720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.672714] LNet: Removed LNI 192.168.203.158@tcp [ 911.661914] Key type .llcrypt unregistered [ 911.666426] Key type ._llcrypt unregistered [ 927.766935] Key type ._llcrypt registered [ 927.771519] Key type .llcrypt registered [ 927.870858] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 928.914712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.947098] alg: No test for adler32 (adler32-zlib) [ 929.943790] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 930.194995] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 931.896163] Key type lgssc registered [ 932.808772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.536440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.559764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.100089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.252047] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 951.267320] LustreError: 34965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.281755] LustreError: 34965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 952.469032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.747075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.599984] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 965.868439] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 971.779426] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 977.743334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.981110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.257775] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 986.166316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 989.735302] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.585077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.588595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.600895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.635866] Lustre: Skipped 1 previous similar message [ 1012.718528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.726083] Lustre: Skipped 2 previous similar messages [ 1013.756843] Lustre: server umount lustre-OST0000 complete [ 1019.361133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1019.368422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.390336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.946038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.946380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.988238] Lustre: Skipped 1 previous similar message [ 1024.387735] Lustre: server umount lustre-MDT0000 complete [ 1028.082290] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.098805] LustreError: 34962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1028.137053] LustreError: 34945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541502 with bad export cookie 10189221849394797056 [ 1028.154579] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.553681] Lustre: server umount lustre-MDT0001 complete [ 1040.762558] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1043.713459] Key type lgssc unregistered [ 1044.014570] LNet: 38522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.020040] LNetError: 38522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.042504] LNet: Removed LNI 192.168.203.158@tcp [ 1044.933208] Key type .llcrypt unregistered [ 1044.937195] Key type ._llcrypt unregistered [ 1067.241409] Key type ._llcrypt registered [ 1067.244449] Key type .llcrypt registered [ 1067.356438] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1068.159867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.205124] alg: No test for adler32 (adler32-zlib) [ 1069.338428] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1069.574194] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1071.266375] Key type lgssc registered [ 1072.417470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1079.304265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1079.314467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.801040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.977406] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1091.057649] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.082749] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1092.861506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.142593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.032693] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1106.449918] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1113.557631] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1121.668674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.227099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.334540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1129.289569] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1137.674992] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.637424] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1146.761780] Lustre: server umount lustre-MDT0000 complete [ 1148.902161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.905161] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.925723] Lustre: Skipped 2 previous similar messages [ 1148.942482] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1150.168795] LustreError: 39914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541624 with bad export cookie 10201692092606061998 [ 1150.181827] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.725608] LustreError: 39930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.771537] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 1154.017296] LustreError: 41507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.032661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.053193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1156.549480] Lustre: server umount lustre-MDT0001 complete [ 1185.249928] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1185.532920] Lustre: server umount lustre-OST0000 complete [ 1200.295350] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1202.646328] Key type lgssc unregistered [ 1202.974986] LNet: 43802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.982850] LNetError: 43802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.995991] LNet: Removed LNI 192.168.203.158@tcp [ 1203.720916] Key type .llcrypt unregistered [ 1203.723298] Key type ._llcrypt unregistered [ 1223.218834] Key type ._llcrypt registered [ 1223.223794] Key type .llcrypt registered [ 1223.345178] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1224.421270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.442158] alg: No test for adler32 (adler32-zlib) [ 1225.435475] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1225.622777] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1227.272967] Key type lgssc registered [ 1228.369299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.327960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.346045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.685494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.265064] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1247.202084] LustreError: 45195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.212479] LustreError: 45195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1250.837923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.259171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.773369] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1263.007995] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1268.415484] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1274.254025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1274.635072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.284471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1282.800217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1286.713812] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.282775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1293.291260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.296736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1293.321525] Lustre: Skipped 1 previous similar message [ 1294.623420] Lustre: lustre-OST0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1294.627731] Lustre: Skipped 1 previous similar message [ 1298.405864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.412719] Lustre: Skipped 1 previous similar message [ 1303.527964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.542270] Lustre: Skipped 2 previous similar messages [ 1307.104782] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1307.282928] Lustre: server umount lustre-OST0000 complete [ 1328.098231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1328.107224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.123755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1328.126729] Lustre: Skipped 1 previous similar message [ 1329.140070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.082719] Lustre: server umount lustre-MDT0000 complete [ 1333.293688] LustreError: 45189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.331318] LustreError: 45189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1334.500210] LustreError: 45174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541808 with bad export cookie 2721086460655878425 [ 1334.500760] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.517024] LustreError: 45174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1341.038253] Lustre: server umount lustre-MDT0001 complete [ 1350.842485] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1353.621553] Key type lgssc unregistered [ 1353.855242] LNet: 48907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.866840] LNetError: 48907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.898588] LNet: Removed LNI 192.168.203.158@tcp [ 1354.736674] Key type .llcrypt unregistered [ 1354.740175] Key type ._llcrypt unregistered [ 1377.873792] Key type ._llcrypt registered [ 1377.885159] Key type .llcrypt registered [ 1378.018515] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1379.148245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.168431] alg: No test for adler32 (adler32-zlib) [ 1380.249471] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1380.508034] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1382.169467] Key type lgssc registered [ 1383.223916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1390.663990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1390.683149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.161790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.716131] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1402.345773] LustreError: 50321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.370202] LustreError: 50321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1404.683052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.037248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.071766] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1416.074776] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1421.310527] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1426.864808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.235791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.012983] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1437.521752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1440.168621] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1445.986314] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1449.621329] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1462.525745] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0413h12m13s [ 1463.905800] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1466.341112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.353268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.376357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.446836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.452041] Lustre: Skipped 4 previous similar messages [ 1468.714499] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 1472.073613] Lustre: server umount lustre-MDT0000 complete [ 1473.507805] LustreError: 50321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.524313] LustreError: 50321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1474.820747] LustreError: 51054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.197227] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1479.662167] LustreError: 50322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.677117] LustreError: 50322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1482.165669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.320507] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.673674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.736487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1483.709426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1487.529528] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1490.551510] Lustre: DEBUG MARKER: Wait for df (36700) ... [ 1491.843554] Lustre: DEBUG MARKER: done [ 1496.034040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.045559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.061402] Lustre: Skipped 2 previous similar messages [ 1496.069353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1496.073467] Lustre: Skipped 1 previous similar message [ 1498.085953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1498.094551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.430110] Lustre: server umount lustre-OST0000 complete [ 1509.347072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.350213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.356528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.356537] Lustre: Skipped 3 previous similar messages [ 1509.366524] Lustre: Skipped 1 previous similar message [ 1510.891727] Lustre: server umount lustre-MDT0000 complete [ 1513.564377] LustreError: 50302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788541987 with bad export cookie 13919037010944869829 [ 1513.571757] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.573493] LustreError: 50302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1513.878853] Lustre: server umount lustre-MDT0001 complete [ 1521.819412] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1524.187320] Key type lgssc unregistered [ 1524.650377] LNet: 55249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.658596] LNetError: 55249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.694465] LNet: Removed LNI 192.168.203.158@tcp [ 1525.668648] Key type .llcrypt unregistered [ 1525.671566] Key type ._llcrypt unregistered [ 1567.712897] Key type ._llcrypt registered [ 1567.719921] Key type .llcrypt registered [ 1567.883240] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1569.425837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.553468] alg: No test for adler32 (adler32-zlib) [ 1571.173661] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1571.627544] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1573.561946] Key type lgssc registered [ 1574.966568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1586.184873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1586.214203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.511744] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1587.553326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1587.784733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.451947] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1600.632692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.849392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1600.900769] Lustre: 57452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1601.130624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.206828] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1613.325362] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1618.800405] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1625.218247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.455700] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1625.637185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.594760] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1633.437469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1638.876038] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1644.768983] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0413h15m17s [ 1646.151855] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1652.360489] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1654.844481] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1654.913637] LustreError: 57460:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1675.047503] Lustre: lustre-MDT0000: Client 660b4349-1e67-4505-b5c3-d56b981d3b0f (at 192.168.203.58@tcp) reconnecting [ 1695.009243] LustreError: 57460:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1697.324815] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1699.502581] Lustre: DEBUG MARKER: done [ 1705.446246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.450673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1705.451483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1705.474910] Lustre: Skipped 1 previous similar message [ 1707.379929] Lustre: server umount lustre-OST0000 complete [ 1713.635522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1713.648646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.673840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.682092] Lustre: Skipped 1 previous similar message [ 1714.722108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.732345] Lustre: Skipped 2 previous similar messages [ 1716.783099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1716.797246] Lustre: Skipped 1 previous similar message [ 1717.059387] Lustre: server umount lustre-MDT0000 complete [ 1720.820161] LustreError: 60214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542194 with bad export cookie 14254882293547755426 [ 1720.823241] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.836965] LustreError: 60214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1721.148792] Lustre: server umount lustre-MDT0001 complete [ 1730.607599] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 1733.208201] Key type lgssc unregistered [ 1733.532319] LNet: 61782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.541737] LNetError: 61782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.574150] LNet: Removed LNI 192.168.203.158@tcp [ 1734.376045] Key type .llcrypt unregistered [ 1734.379605] Key type ._llcrypt unregistered [ 1777.330825] Key type ._llcrypt registered [ 1777.333839] Key type .llcrypt registered [ 1777.428643] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 1778.402047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.600583] alg: No test for adler32 (adler32-zlib) [ 1779.703935] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 1780.031032] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 1781.713563] Key type lgssc registered [ 1782.911864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.319334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.342234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.694669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1787.728627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1788.011442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.066916] Lustre: Failing over lustre-MDT0000 [ 1789.516790] Lustre: server umount lustre-MDT0000 complete [ 1796.175820] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:17:48 (1788542268) [ 1803.079229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.614255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.099101] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1816.411608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.536195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1816.581146] Lustre: 64584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.822439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.206776] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1828.185381] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1834.163575] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1842.200242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.618946] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1842.924733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.971883] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1852.655296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1858.062976] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1864.119546] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1873.392453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.397401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.410081] Lustre: Skipped 2 previous similar messages [ 1873.410369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.441562] Lustre: Skipped 2 previous similar messages [ 1876.473278] Lustre: server umount lustre-MDT0000 complete [ 1880.964932] LustreError: 64574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542354 with bad export cookie 17786013900950797894 [ 1880.976497] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.991600] LustreError: 64574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1881.347355] Lustre: server umount lustre-MDT0001 complete [ 1884.042502] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1887.711608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1896.499169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.000484] Lustre: 63507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542357/real 1788542357] req@00000000eccba8d7 x1875422481523328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1900.067781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.088143] Lustre: 63507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542359/real 1788542359] req@000000002eeb727b x1875422481523456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1904.096769] Lustre: 63505:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542362/real 1788542362] req@0000000067df7443 x1875422481523584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1905.256718] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf6d49d6898550729 [ 1905.273521] Lustre: MGC192.168.203.158@tcp: Connection restored to 0@lo (at 0@lo) [ 1905.850065] LustreError: 66432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.884874] LustreError: 66432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1905.987328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.128110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1906.140375] Lustre: 63504:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542364/real 1788542364] req@00000000cd8e7818 x1875422481523840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1908.962766] LustreError: 66433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.042763] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1910.065255] LustreError: 68354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.081233] LustreError: 68354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1911.072566] Lustre: 63506:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542369/real 1788542369] req@00000000c9e9942d x1875422481523968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1915.361855] LustreError: 68353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.384040] LustreError: 68353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1918.110598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.627119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.401235] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1924.074063] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1928.180332] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1931.026168] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1936.352678] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.524890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.529117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.543178] Lustre: Skipped 2 previous similar messages [ 1943.552479] Lustre: Skipped 2 previous similar messages [ 1948.641844] LustreError: 68354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.654112] LustreError: 68354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1948.666173] Lustre: server umount lustre-MDT0000 complete [ 1952.303423] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542426 with bad export cookie 17786013900950800169 [ 1952.328611] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.331396] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1952.727383] Lustre: server umount lustre-MDT0001 complete [ 1959.339930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1968.762943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.120138] Lustre: 63506:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788542427/real 1788542427] req@000000004ac9756b x1875422481550208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788542443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.153942] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.390039] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf6d49d6898550c8c [ 1977.410706] Lustre: MGC192.168.203.158@tcp: Connection restored to 0@lo (at 0@lo) [ 1977.687028] LustreError: 66432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.756727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.813442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1982.202842] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1989.241980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.893074] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 1994.741165] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.528429] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2005.861967] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2014.180792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.182958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.216281] Lustre: Skipped 2 previous similar messages [ 2014.225293] Lustre: Skipped 1 previous similar message [ 2018.568901] Lustre: server umount lustre-MDT0000 complete [ 2019.298390] LustreError: 71033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.318148] LustreError: 71033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2021.658670] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542495 with bad export cookie 17786013900950801548 [ 2021.659815] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.671228] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2021.942050] Lustre: server umount lustre-MDT0001 complete [ 2027.540060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2030.543703] Lustre: DEBUG MARKER: files compared the same [ 2042.662954] Lustre: server umount lustre-OST0000 complete [ 2058.684640] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2060.975498] Key type lgssc unregistered [ 2061.224745] LNet: 74309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.234727] LNetError: 74309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.273142] LNet: Removed LNI 192.168.203.158@tcp [ 2062.144418] Key type .llcrypt unregistered [ 2062.146928] Key type ._llcrypt unregistered [ 2082.738474] Key type ._llcrypt registered [ 2082.741660] Key type .llcrypt registered [ 2082.841953] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2083.766276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.870557] alg: No test for adler32 (adler32-zlib) [ 2084.973861] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2085.241509] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2086.920522] Key type lgssc registered [ 2088.087281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.947777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.993928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.531536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.702488] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 2107.873777] LustreError: 75703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.885778] LustreError: 75703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2109.796819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.163742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2114.157305] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 2121.278580] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.073288] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2133.796974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.069702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.980652] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug +malloc all [ 2142.255887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2147.158612] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2155.489210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2155.501788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.522027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.003562] Lustre: server umount lustre-OST0000 complete [ 2161.640523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2161.644206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.673914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.656741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.785133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.798767] Lustre: Skipped 3 previous similar messages [ 2169.934171] Lustre: server umount lustre-MDT0000 complete [ 2172.915943] LustreError: 75703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.948125] LustreError: 75703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2173.859726] LustreError: 75683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542647 with bad export cookie 8616266999923576668 [ 2173.875900] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.280465] Lustre: server umount lustre-MDT0001 complete [ 2186.084315] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2188.987966] Key type lgssc unregistered [ 2189.382563] LNet: 79233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.389625] LNetError: 79233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.428502] LNet: Removed LNI 192.168.203.158@tcp [ 2190.310702] Key type .llcrypt unregistered [ 2190.317090] Key type ._llcrypt unregistered [ 2209.701793] Key type ._llcrypt registered [ 2209.704295] Key type .llcrypt registered [ 2209.772934] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2210.611734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2210.727471] alg: No test for adler32 (adler32-zlib) [ 2211.746140] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2211.944261] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2213.608234] Key type lgssc registered [ 2214.608427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.193091] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2221.217492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2236.768399] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2241.888507] LustreError: 80599:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.158@tcp: failed processing log, type 4: rc = -110 [ 2268.576183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.924453] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2281.570759] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2287.897708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.004545] Lustre: *** cfs_fail_loc=706, val=0*** [ 2288.481231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.547749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2292.535869] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2293.734577] LustreError: 80625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.751397] LustreError: 80625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2298.850833] LustreError: 80624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.947950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.474229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2305.027987] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2312.538237] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.278655] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2320.867043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.883422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.895418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.942129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.966253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.707922] Lustre: server umount lustre-OST0000 complete [ 2331.109112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.116218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.146507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.153262] Lustre: Skipped 1 previous similar message [ 2334.177598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.179931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.340175] Lustre: server umount lustre-MDT0000 complete [ 2339.276064] LustreError: 80606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788542813 with bad export cookie 12078854632518843015 [ 2339.281251] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.293996] LustreError: 80606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2339.638248] Lustre: server umount lustre-MDT0001 complete [ 2349.456801] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2352.077814] Key type lgssc unregistered [ 2352.376708] LNet: 84213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.391636] LNetError: 84213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.422701] LNet: Removed LNI 192.168.203.158@tcp [ 2353.284656] Key type .llcrypt unregistered [ 2353.290093] Key type ._llcrypt unregistered [ 2373.764592] Key type ._llcrypt registered [ 2373.771344] Key type .llcrypt registered [ 2373.899225] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2374.784140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.855488] alg: No test for adler32 (adler32-zlib) [ 2375.890743] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2376.108731] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2377.784230] Key type lgssc registered [ 2378.990593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.310209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.329863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.730991] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2395.895798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.362984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2400.812450] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2401.765212] LustreError: 86292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.799643] LustreError: 86292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2406.883763] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.914853] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2408.590977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.866126] LustreError: 86292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.896696] LustreError: 86292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2408.916323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.051536] LustreError: 86295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.064608] LustreError: 86295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2414.289677] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2419.172513] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.187146] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2420.618796] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.627631] Lustre: MGS: Client a27409d0-0efd-4392-8f75-64b3e10c18a5 (at 0@lo) reconnecting [ 2420.646091] LustreError: 86306:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2420.835948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.900607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2423.946927] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2436.065843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.077847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.096188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2437.602364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2437.604451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.404900] Lustre: server umount lustre-OST0000 complete [ 2447.843083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.849989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.857199] Lustre: Skipped 1 previous similar message [ 2447.859026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2447.863869] Lustre: Skipped 2 previous similar messages [ 2451.133107] Lustre: server umount lustre-MDT0000 complete [ 2452.962541] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.979439] LustreError: 86296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2454.552978] Lustre: server umount lustre-MDT0001 complete [ 2457.665622] Lustre: server umount MGS complete [ 2467.477632] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:29:00 (1788542940) [ 2491.836817] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 2499.515907] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2530.820820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.361995] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2543.255711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.491397] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2543.551753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2543.629875] Lustre: lustre-MDT0001: new disk, initializing [ 2543.680300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.694092] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2543.698557] Lustre: 93462:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2543.706479] LustreError: 93462:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2547.053797] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2555.037426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.252496] Lustre: lustre-OST0000: new disk, initializing [ 2555.255471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2555.261627] Lustre: 94226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2555.313411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.642210] Lustre: 94258:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000413a30f8 x1875423107520000/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788543046 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2560.445367] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2568.237805] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2573.288991] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.307219] Lustre: MGS: Client 4e3646b9-981c-4920-baa1-139e77d30614 (at 0@lo) reconnecting [ 2573.333668] LustreError: 93448:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2573.364861] Lustre: 92730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2573.399953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2573.471875] Lustre: lustre-MDT0000: new disk, initializing [ 2573.583348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.600275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2573.614044] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2574.627395] Lustre: 95295:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e9fb80b2 x1875423107527040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788543064 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2576.915564] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2579.695951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2579.706128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2579.841310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2599.907058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2599.917439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.932367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.540194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.552399] Lustre: Skipped 3 previous similar messages [ 2605.771663] Lustre: server umount lustre-OST0000 complete [ 2610.661762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.662934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2610.673105] Lustre: Skipped 1 previous similar message [ 2610.676532] LustreError: Skipped 1 previous similar message [ 2615.020681] Lustre: server umount lustre-MDT0000 complete [ 2615.781037] LustreError: 93433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.794259] LustreError: 93433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2624.273918] Lustre: server umount lustre-MDT0001 complete [ 2627.494875] Lustre: server umount MGS complete [ 2637.590877] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:31:50 (1788543110) [ 2652.950751] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2655.853647] Key type lgssc unregistered [ 2656.179824] LNet: 97789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.188123] LNetError: 97789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.212949] LNet: Removed LNI 192.168.203.158@tcp [ 2656.974413] Key type .llcrypt unregistered [ 2656.979231] Key type ._llcrypt unregistered [ 2672.843200] Key type ._llcrypt registered [ 2672.845151] Key type .llcrypt registered [ 2672.986789] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2673.683393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.732976] alg: No test for adler32 (adler32-zlib) [ 2674.794250] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2675.004296] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2676.672176] Key type lgssc registered [ 2677.916639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2684.925796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2684.955802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.638179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.400548] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2692.066109] LustreError: 99007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.089981] LustreError: 99007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2696.161939] LustreError: 99008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.301132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.728127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.757253] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2711.753379] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.304779] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2723.312235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.595075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.018981] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2731.818682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2735.524480] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2742.251611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.252509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.257334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.316128] Lustre: Skipped 1 previous similar message [ 2747.365114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2747.380157] Lustre: Skipped 2 previous similar messages [ 2748.109597] Lustre: server umount lustre-OST0000 complete [ 2751.979935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2751.993823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.017460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.601649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.614616] Lustre: Skipped 3 previous similar messages [ 2762.740977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.749143] Lustre: Skipped 1 previous similar message [ 2766.304146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.607747] Lustre: server umount lustre-MDT0000 complete [ 2767.842726] LustreError: 99008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.859083] LustreError: 99008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2769.728687] LustreError: 98988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543243 with bad export cookie 5998451059972826586 [ 2769.731491] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.739229] LustreError: 98988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2770.279254] Lustre: server umount lustre-MDT0001 complete [ 2782.174863] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2784.475214] Key type lgssc unregistered [ 2784.731551] LNet: 102540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.743543] LNetError: 102540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.759397] LNet: Removed LNI 192.168.203.158@tcp [ 2785.636856] Key type .llcrypt unregistered [ 2785.641380] Key type ._llcrypt unregistered [ 2799.655860] Key type ._llcrypt registered [ 2799.659249] Key type .llcrypt registered [ 2799.749408] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2800.549734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2800.581727] alg: No test for adler32 (adler32-zlib) [ 2801.508941] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2801.733328] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2803.400567] Key type lgssc registered [ 2804.597407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2809.773104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2809.777991] LustreError: 103610:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2809.780785] LustreError: 103611:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2809.787474] LustreError: 103610:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2809.792969] LustreError: 103610:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2809.797233] LustreError: 103610:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2809.812726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.207866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.464731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.777240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2823.273844] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2828.196145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing lsmod [ 2830.844527] LustreError: 104809:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2830.886862] LustreError: 104816:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2830.895343] LustreError: 104809:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 2830.900856] LustreError: 104809:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2830.914123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.915293] LustreError: 104809:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2831.283031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.401953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2834.416872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.432630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.455252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.471290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.492214] Lustre: Skipped 1 previous similar message [ 2841.574106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.585077] Lustre: Skipped 4 previous similar messages [ 2843.628800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.635536] Lustre: Skipped 2 previous similar messages [ 2848.224113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2848.436164] Lustre: server umount lustre-MDT0000 complete [ 2848.739056] LustreError: 105001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.748738] LustreError: 105001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2851.775980] LustreError: 103628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543325 with bad export cookie 16730325259665904714 [ 2851.794240] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.801794] LustreError: 103628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2851.873269] LustreError: 104079:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bb4b0898 x1875423553147776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2852.220969] Lustre: server umount lustre-MDT0001 complete [ 2866.168427] Lustre: server umount lustre-OST0000 complete [ 2874.009685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.573523] LustreError: 105882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.645910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.110386] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2879.974406] LustreError: 105883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.070038] LustreError: 105882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.934385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.242473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2892.101668] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2898.337569] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2903.863738] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2910.715484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2911.156775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.279605] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 2919.365196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2925.698497] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2933.730967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2933.739164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.758051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.758265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2936.429288] Lustre: server umount lustre-OST0000 complete [ 2939.884289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.898458] Lustre: Skipped 1 previous similar message [ 2944.490396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2944.493716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.514203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.516995] Lustre: Skipped 3 previous similar messages [ 2945.858926] Lustre: server umount lustre-MDT0000 complete [ 2949.553346] LustreError: 105863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543423 with bad export cookie 16730325259665905414 [ 2949.566974] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2949.579714] LustreError: 105863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2949.913105] Lustre: server umount lustre-MDT0001 complete [ 2960.131898] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 2962.708712] Key type lgssc unregistered [ 2963.032924] LNet: 109416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.041521] LNetError: 109416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2963.059911] LNet: Removed LNI 192.168.203.158@tcp [ 2963.934418] Key type .llcrypt unregistered [ 2963.938786] Key type ._llcrypt unregistered [ 2983.558364] Key type ._llcrypt registered [ 2983.561529] Key type .llcrypt registered [ 2983.649745] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 2984.606284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2984.697563] alg: No test for adler32 (adler32-zlib) [ 2985.882036] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 2986.204201] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 2987.904615] Key type lgssc registered [ 2989.461789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.507142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.529994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.930393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.871588] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3008.492191] LustreError: 110806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.516374] LustreError: 110806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3010.573985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.043488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3015.075226] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3021.344826] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3025.558476] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3030.831573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.178126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.010684] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3041.886541] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3043.512292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3050.815477] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3054.664580] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3059.179852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.181560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.194463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.210040] Lustre: Skipped 1 previous similar message [ 3059.218095] Lustre: Skipped 1 previous similar message [ 3060.393061] Lustre: server umount lustre-OST0000 complete [ 3064.289754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.305312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.316253] Lustre: Skipped 1 previous similar message [ 3069.409809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.418055] Lustre: Skipped 1 previous similar message [ 3070.255789] Lustre: server umount lustre-MDT0000 complete [ 3073.797333] LustreError: 110787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543547 with bad export cookie 4944180262312965789 [ 3073.798676] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3073.818376] LustreError: 110787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3074.203965] Lustre: server umount lustre-MDT0001 complete [ 3084.644555] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3086.996705] Key type lgssc unregistered [ 3087.213233] LNet: 114444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.227063] LNetError: 114444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3087.245827] LNet: Removed LNI 192.168.203.158@tcp [ 3087.943211] Key type .llcrypt unregistered [ 3087.946837] Key type ._llcrypt unregistered [ 3103.680191] Key type ._llcrypt registered [ 3103.684374] Key type .llcrypt registered [ 3103.808434] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3105.370659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3105.402093] alg: No test for adler32 (adler32-zlib) [ 3106.486620] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 3106.733813] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3108.376352] Key type lgssc registered [ 3109.639158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3117.095902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3117.115955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.657713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.992729] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3128.813293] LustreError: 115665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.829335] LustreError: 115665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3130.415086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.735351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.794685] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3140.673818] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3145.252046] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3151.563197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.045649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.947591] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3163.329790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3167.008903] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3175.905276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.909618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.915546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.933990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3176.938928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.960664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.986511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.996756] Lustre: Skipped 1 previous similar message [ 3180.344578] Lustre: server umount lustre-OST0000 complete [ 3184.106716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.110698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.117298] Lustre: Skipped 1 previous similar message [ 3189.218228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.231516] Lustre: Skipped 2 previous similar messages [ 3189.988489] Lustre: server umount lustre-MDT0000 complete [ 3193.910928] LustreError: 115646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543667 with bad export cookie 7788174481430122933 [ 3193.921146] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.922861] LustreError: 115646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3194.339494] Lustre: server umount lustre-MDT0001 complete [ 3206.527473] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3209.238526] Key type lgssc unregistered [ 3209.502422] LNet: 119195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3209.510110] LNetError: 119195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3209.527961] LNet: Removed LNI 192.168.203.158@tcp [ 3210.411669] Key type .llcrypt unregistered [ 3210.415751] Key type ._llcrypt unregistered [ 3232.323699] Key type ._llcrypt registered [ 3232.325327] Key type .llcrypt registered [ 3232.424981] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3233.071886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3233.107039] alg: No test for adler32 (adler32-zlib) [ 3234.171139] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 3234.356987] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3236.008177] Key type lgssc registered [ 3237.422887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.421324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3244.438463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.791113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.981375] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3255.269458] LustreError: 120607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.288867] LustreError: 120607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3257.415541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.686737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.336856] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3267.329929] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3272.575687] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3278.432816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.705975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.335265] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3291.024251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3291.783152] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.666633] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3302.087128] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3302.097480] Lustre: Skipped 2 previous similar messages [ 3305.858961] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3305.863654] Lustre: Skipped 2 previous similar messages [ 3306.134085] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3306.169674] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3311.747770] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.58@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3311.759690] Lustre: Skipped 2 previous similar messages [ 3314.838336] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.58@tcp 192.168.0.[2,10]@tcp [ 3319.397143] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.58/24@tcp in log lustre-MDT0000 [ 3319.413992] Lustre: Skipped 2 previous similar messages [ 3323.991902] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.58/24@tcp [ 3324.001740] Lustre: Skipped 1 previous similar message [ 3332.067112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.073337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3332.097800] Lustre: Skipped 1 previous similar message [ 3332.113404] Lustre: Skipped 1 previous similar message [ 3336.160787] Lustre: server umount lustre-OST0000 complete [ 3342.307237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.310710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.332328] Lustre: Skipped 1 previous similar message [ 3342.340291] Lustre: Skipped 1 previous similar message [ 3346.935681] Lustre: server umount lustre-MDT0000 complete [ 3347.428150] LustreError: 120607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.448061] LustreError: 120607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3350.639641] LustreError: 120587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543824 with bad export cookie 4182879438036988118 [ 3350.639911] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.647129] LustreError: 120587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3351.118550] Lustre: server umount lustre-MDT0001 complete [ 3361.168923] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3363.520890] Key type lgssc unregistered [ 3363.828663] LNet: 125395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3363.833483] LNetError: 125395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3363.860722] LNet: Removed LNI 192.168.203.158@tcp [ 3364.642642] Key type .llcrypt unregistered [ 3364.644793] Key type ._llcrypt unregistered [ 3390.816690] Key type ._llcrypt registered [ 3390.823278] Key type .llcrypt registered [ 3390.940200] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3392.282570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3392.317228] alg: No test for adler32 (adler32-zlib) [ 3393.328790] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 3393.541526] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3395.192160] Key type lgssc registered [ 3396.116861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3402.109899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3402.123092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.480365] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3403.493601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.506827] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.58/24@tcp [ 3406.638387] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3413.993820] LustreError: 126952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.055121] LustreError: 126952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3417.040914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.522734] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3417.599876] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.58/24@tcp [ 3417.692713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3422.435924] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3429.855357] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3435.081803] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3441.232983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.898439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.002377] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3449.091240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3456.274860] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3468.771934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.782891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.800709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.797614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.798210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.807149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.821620] Lustre: Skipped 1 previous similar message [ 3471.515583] Lustre: server umount lustre-OST0000 complete [ 3474.917980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.945535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.953616] Lustre: Skipped 1 previous similar message [ 3480.034528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.042724] Lustre: Skipped 1 previous similar message [ 3485.157175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.169766] Lustre: Skipped 1 previous similar message [ 3489.250869] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3489.377816] Lustre: server umount lustre-MDT0000 complete [ 3490.275282] LustreError: 126946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.288242] LustreError: 126946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3491.980599] LustreError: 126933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788543965 with bad export cookie 12140901564850210527 [ 3491.991353] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3491.992105] LustreError: 126933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3492.219835] Lustre: server umount lustre-MDT0001 complete [ 3501.139577] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3503.893228] Key type lgssc unregistered [ 3504.123089] LNet: 130540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3504.129710] LNetError: 130540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3504.144603] LNet: Removed LNI 192.168.203.158@tcp [ 3505.102346] Key type .llcrypt unregistered [ 3505.107909] Key type ._llcrypt unregistered [ 3528.251754] Key type ._llcrypt registered [ 3528.254615] Key type .llcrypt registered [ 3528.367430] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3543.669677] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3544.468851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3544.489069] alg: No test for adler32 (adler32-zlib) [ 3545.690373] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 3546.034802] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3547.696168] Key type lgssc registered [ 3548.970432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3580.781058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3580.804428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.121192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3582.196134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3582.337535] Lustre: lustre-MDT0000: new disk, initializing [ 3582.426934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.448252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3586.690898] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3596.467964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.566349] Lustre: 133937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3596.631969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3596.635558] Lustre: Skipped 1 previous similar message [ 3596.705969] Lustre: lustre-MDT0001: new disk, initializing [ 3596.799286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3596.835389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3596.850683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3601.654681] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3611.346250] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.619969] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3623.563143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.899167] Lustre: lustre-OST0000: new disk, initializing [ 3623.902232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3623.917490] Lustre: 135905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3624.028528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.747559] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3633.174364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3633.182660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3633.246231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3638.176360] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3643.813273] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3644.115754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3649.736538] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3650.095518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3653.608505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3653.617966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.620433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3653.637113] Lustre: Skipped 1 previous similar message [ 3653.649572] Lustre: Skipped 1 previous similar message [ 3658.722778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.729783] Lustre: Skipped 1 previous similar message [ 3663.846940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.862485] Lustre: Skipped 1 previous similar message [ 3666.400115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3666.622941] Lustre: server umount lustre-OST0000 complete [ 3673.569214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3673.584947] LustreError: Skipped 1 previous similar message [ 3673.589383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.598783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.579709] Lustre: server umount lustre-MDT0000 complete [ 3679.445416] LustreError: 134732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544153 with bad export cookie 2913583766490052767 [ 3679.447613] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.461554] LustreError: 134732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3679.749679] Lustre: server umount lustre-MDT0001 complete [ 3687.197793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.644140] LustreError: 138122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.664970] LustreError: 138122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3687.716966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.749665] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3693.029851] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.124732] LustreError: 138122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.041527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.368370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3703.806837] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3710.705947] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3716.638312] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3724.079512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.268397] LustreError: 139941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.507049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.656403] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3737.041538] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3741.388156] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3741.608715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3745.843128] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3746.111195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3754.795344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3754.982547] Lustre: lustre-OST0001: new disk, initializing [ 3754.985637] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3754.991312] Lustre: 141497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3755.074329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.800442] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3761.192212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3761.200930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3761.290671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3767.800139] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3767.977114] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3771.539278] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3771.717685] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3776.399667] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c5cd8199000.ost_server_uuid 50 [ 3777.911459] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c5cd8199000.ost_server_uuid in IDLE state after 0 sec [ 3786.730876] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.739886] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.744624] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3786.744633] Lustre: Skipped 2 previous similar messages [ 3786.754319] Lustre: Skipped 2 previous similar messages [ 3786.759903] LustreError: Skipped 1 previous similar message [ 3786.899499] Lustre: server umount lustre-OST0001 complete [ 3791.332943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3791.344468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.855410] LustreError: 139942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.875534] LustreError: 139942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3796.548413] Lustre: server umount lustre-OST0000 complete [ 3802.081084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3802.085304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.093973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.100650] Lustre: Skipped 2 previous similar messages [ 3802.120900] Lustre: Skipped 4 previous similar messages [ 3815.401531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3815.495714] Lustre: server umount lustre-MDT0000 complete [ 3816.418392] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.430758] LustreError: 138121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3818.487184] LustreError: 141496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544292 with bad export cookie 2913583766490053649 [ 3818.496277] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.506283] LustreError: 141496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3818.732162] Lustre: server umount lustre-MDT0001 complete [ 3833.335272] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 3835.876367] Key type lgssc unregistered [ 3836.106522] LNet: 144495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3836.113866] LNetError: 144495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.132576] LNet: Removed LNI 192.168.203.158@tcp [ 3836.818181] Key type .llcrypt unregistered [ 3836.821114] Key type ._llcrypt unregistered [ 3871.006718] Key type ._llcrypt registered [ 3871.009677] Key type .llcrypt registered [ 3871.068915] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 3885.336058] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 3886.089395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3886.099941] alg: No test for adler32 (adler32-zlib) [ 3887.073929] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 3887.274937] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 3888.944770] Key type lgssc registered [ 3889.892288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3917.672301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3917.686397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.860182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3918.894414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3918.962927] Lustre: lustre-MDT0000: new disk, initializing [ 3919.024426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.036276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3922.297041] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3930.947612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.052150] Lustre: 148646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.131869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3931.136237] Lustre: Skipped 1 previous similar message [ 3931.245350] Lustre: lustre-MDT0001: new disk, initializing [ 3931.404070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.483059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3931.505746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3935.741765] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3945.739597] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.484902] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3957.916938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.174388] Lustre: lustre-OST0000: new disk, initializing [ 3958.176640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3958.180504] Lustre: 150615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3958.235931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.407391] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 3965.492080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3965.496886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3965.582575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3971.809649] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3975.812971] Lustre: Failing over lustre-OST0000 [ 3976.060792] Lustre: server umount lustre-OST0000 complete [ 3993.056834] Lustre: 146869:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788544450/real 1788544450] req@00000000c7c2e608 x1875424691912960/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788544466 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3993.061079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.083671] Lustre: 146869:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3994.422295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.773823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.830174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.856340] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3994.964144] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3996.193562] Lustre: 146869:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788544454/real 1788544454] req@00000000d81d50ec x1875424691913472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788544470 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4001.923860] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4001.956897] Lustre: 146871:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788544459/real 1788544459] req@000000004ba82722 x1875424691915264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788544475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4001.982525] Lustre: 146871:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4004.047611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4004.047759] Lustre: lustre-OST0000: Recovery over after 0:10, of 3 clients 3 recovered and 0 were evicted. [ 4004.968548] Lustre: Failing over lustre-MDT0000 [ 4005.277307] Lustre: server umount lustre-MDT0000 complete [ 4007.008114] Lustre: 146870:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788544464/real 1788544464] req@0000000030768673 x1875424691916416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788544480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4007.022289] Lustre: 146870:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4008.422987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.431508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.244096] LustreError: 151285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4010.271989] LustreError: 151285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4014.568654] LustreError: 148658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.579029] LustreError: 148658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4019.683262] LustreError: 148655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.713104] LustreError: 148655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4023.418750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.529418] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.801891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.879197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4025.604962] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4027.401471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4028.913811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4028.920862] Lustre: Skipped 1 previous similar message [ 4028.961363] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4029.014520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4034.026702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.030438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4034.040962] Lustre: Skipped 3 previous similar messages [ 4035.979066] Lustre: server umount lustre-OST0000 complete [ 4039.137637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.139195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4039.157218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.157228] Lustre: Skipped 1 previous similar message [ 4044.259326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4044.264949] Lustre: Skipped 2 previous similar messages [ 4045.299583] Lustre: server umount lustre-MDT0000 complete [ 4048.383283] LustreError: 148638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544522 with bad export cookie 13794343423571131333 [ 4048.390747] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.391288] LustreError: 148638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4048.693980] Lustre: server umount lustre-MDT0001 complete [ 4056.958920] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 4059.118653] Key type lgssc unregistered [ 4059.389762] LNet: 154264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.394824] LNetError: 154264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.410505] LNet: Removed LNI 192.168.203.158@tcp [ 4060.424689] Key type .llcrypt unregistered [ 4060.426788] Key type ._llcrypt unregistered [ 4097.270850] Key type ._llcrypt registered [ 4097.274807] Key type .llcrypt registered [ 4097.398120] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4098.262395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.341148] alg: No test for adler32 (adler32-zlib) [ 4099.372722] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 4099.629461] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 4101.312324] Key type lgssc registered [ 4102.348723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4109.134255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4109.160736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.574941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.292104] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4121.059752] LustreError: 155781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.075675] LustreError: 155781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4121.819680] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4124.924169] LustreError: 155809:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4124.935140] LustreError: 155809:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4124.941447] LustreError: 155809:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4125.209781] Lustre: server umount lustre-MDT0000 complete [ 4137.942420] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4146.645818] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4174.909306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.134980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4175.194415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4175.272435] Lustre: lustre-MDT0000: new disk, initializing [ 4175.377914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.408381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4179.612686] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4189.341901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.460494] Lustre: 159108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4189.506806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4189.510794] Lustre: Skipped 1 previous similar message [ 4189.575470] Lustre: lustre-MDT0001: new disk, initializing [ 4189.654808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.709805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4189.723453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4194.104989] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4201.990403] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4207.061480] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4212.526259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.775724] Lustre: lustre-OST0000: new disk, initializing [ 4212.779774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4212.786465] Lustre: 161079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4212.853222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.074275] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4220.490416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4220.513659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4220.619174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4226.473629] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4230.832354] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4232.335646] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:58:25 (1788544705) [ 4445.669861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4445.675884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.685829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.177532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.198723] Lustre: Skipped 1 previous similar message [ 4446.208195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.214901] Lustre: Skipped 2 previous similar messages [ 4450.531438] Lustre: server umount lustre-MDT0000 complete [ 4451.300249] LustreError: 159116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.315158] LustreError: 159116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4453.023740] LustreError: 159101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788544926 with bad export cookie 11575009176599422539 [ 4453.026533] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.032222] LustreError: 159101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4456.416827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.424569] LustreError: 159117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.431427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4458.464572] LustreError: 162847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.464767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4459.353206] Lustre: server umount lustre-MDT0001 complete [ 4462.150892] LustreError: 163380:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x4:0x0].0x0 (00000000d308cc0a) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4468.422785] Lustre: server umount lustre-OST0000 complete [ 4476.112434] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4483.596260] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4513.611440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.810229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4513.841589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4513.903022] Lustre: lustre-MDT0000: new disk, initializing [ 4513.977243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.990429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4517.667150] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4527.090994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.197102] Lustre: 165744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4527.250595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4527.254283] Lustre: Skipped 1 previous similar message [ 4527.315342] Lustre: lustre-MDT0001: new disk, initializing [ 4527.387362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4527.422549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4527.438828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4531.988531] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4540.777455] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4546.009745] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4552.377359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4552.823993] Lustre: lustre-OST0000: new disk, initializing [ 4552.826575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4552.841540] Lustre: 167712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4552.951736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.779722] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4559.973723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4559.994284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4560.135632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4567.521644] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4571.891203] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4572.099187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4576.383320] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4576.566960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4580.838746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.841359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.857103] LustreError: Skipped 1 previous similar message [ 4580.858785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.868926] Lustre: Skipped 1 previous similar message [ 4584.610163] Lustre: server umount lustre-OST0000 complete [ 4589.029196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.035735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.050575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.058825] Lustre: Skipped 1 previous similar message [ 4602.339248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4602.546147] Lustre: server umount lustre-MDT0000 complete [ 4605.409457] LustreError: 165755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.424562] LustreError: 165755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4606.240763] LustreError: 165737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545080 with bad export cookie 11575009176599581369 [ 4606.247093] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4606.263647] LustreError: 165737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4606.681556] Lustre: server umount lustre-MDT0001 complete [ 4614.447296] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:04:47 (1788545087) [ 4628.780396] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4636.860774] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4668.076724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.275686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4668.332615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4668.428480] Lustre: lustre-MDT0000: new disk, initializing [ 4668.509938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.538394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4673.011485] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4682.900543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.027587] Lustre: 172467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4683.345076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4687.397898] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4696.215156] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4701.633289] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4709.024417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.304253] Lustre: lustre-OST0000: new disk, initializing [ 4709.307067] Lustre: Skipped 1 previous similar message [ 4709.311926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4709.317884] Lustre: Skipped 2 previous similar messages [ 4709.321524] Lustre: 174446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4715.195044] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4716.661356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4716.671075] Lustre: Skipped 1 previous similar message [ 4716.683605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4716.848036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4723.695868] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.385314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4732.389223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.400168] Lustre: Skipped 1 previous similar message [ 4732.404478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4732.408026] Lustre: Skipped 7 previous similar messages [ 4744.162718] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4744.328897] Lustre: server umount lustre-OST0000 complete [ 4753.793290] Lustre: server umount lustre-MDT0000 complete [ 4757.077863] LustreError: 172460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545231 with bad export cookie 11575009176599582251 [ 4757.086988] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.088921] LustreError: 172460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4757.324087] Lustre: server umount lustre-MDT0001 complete [ 4764.729066] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:07:17 (1788545237) [ 4779.936947] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 4789.128967] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4819.896344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.158502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4820.190502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4820.289695] Lustre: lustre-MDT0000: new disk, initializing [ 4820.350401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4820.357978] Lustre: Skipped 2 previous similar messages [ 4820.385839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4824.941837] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4834.215490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.366545] Lustre: 178768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4834.659443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4838.512842] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4846.604344] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4851.709191] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4857.559752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.830959] Lustre: 180747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4863.837200] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4868.183648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4868.307513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4872.221451] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4878.818366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.821332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.834640] LustreError: Skipped 2 previous similar messages [ 4878.835247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4878.835251] Lustre: Skipped 8 previous similar messages [ 4878.879208] Lustre: Skipped 4 previous similar messages [ 4884.836761] Lustre: server umount lustre-OST0000 complete [ 4894.708857] Lustre: server umount lustre-MDT0000 complete [ 4898.167711] LustreError: 178760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545372 with bad export cookie 11575009176599583336 [ 4898.175277] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.184814] LustreError: 178760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4898.490515] Lustre: server umount lustre-MDT0001 complete [ 4909.146469] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 4911.691285] Key type lgssc unregistered [ 4911.954313] LNet: 182602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4911.963312] LNetError: 182602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4911.977943] LNet: Removed LNI 192.168.203.158@tcp [ 4912.733531] Key type .llcrypt unregistered [ 4912.738552] Key type ._llcrypt unregistered [ 4934.176333] Key type ._llcrypt registered [ 4934.178318] Key type .llcrypt registered [ 4934.278306] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 4935.769550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4935.781087] alg: No test for adler32 (adler32-zlib) [ 4936.824675] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 4937.038287] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 4938.739773] Key type lgssc registered [ 4939.835247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4947.269642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4947.294880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.885727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.248406] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4958.178717] LustreError: 184018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.204210] LustreError: 184018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4961.595797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4961.957589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4965.940310] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4972.705390] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4976.935285] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4983.004877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.298763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.523249] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 4993.531670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4996.005821] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5013.986737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5013.993576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.995546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.025668] Lustre: Skipped 1 previous similar message [ 5019.107745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5019.110326] Lustre: Skipped 1 previous similar message [ 5020.133576] Lustre: server umount lustre-OST0000 complete [ 5024.230071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.232133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.232144] Lustre: Skipped 1 previous similar message [ 5024.261328] Lustre: Skipped 1 previous similar message [ 5029.354297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.363932] Lustre: Skipped 2 previous similar messages [ 5030.192168] Lustre: server umount lustre-MDT0000 complete [ 5034.465452] LustreError: 184745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.482674] LustreError: 184745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5034.576697] LustreError: 183999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545508 with bad export cookie 8353794186199119021 [ 5034.580642] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5034.585116] LustreError: 183999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5034.885597] Lustre: server umount lustre-MDT0001 complete [ 5049.366621] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 5052.569920] Key type lgssc unregistered [ 5052.910548] LNet: 187570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5052.921684] LNetError: 187570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5052.947757] LNet: Removed LNI 192.168.203.158@tcp [ 5054.009349] Key type .llcrypt unregistered [ 5054.012760] Key type ._llcrypt unregistered [ 5075.816093] Key type ._llcrypt registered [ 5075.818040] Key type .llcrypt registered [ 5075.903870] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5076.976828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5077.022360] alg: No test for adler32 (adler32-zlib) [ 5078.025911] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 5078.255908] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 5079.880313] Key type lgssc registered [ 5080.833703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5087.914688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5087.927537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.478242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.742158] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5100.005085] LustreError: 188983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.014067] LustreError: 188983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5101.031893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.295174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5105.704782] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5112.442569] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5117.662488] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5123.465279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5123.779158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.025798] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5135.033892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5136.904478] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5142.497095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5142.506230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5142.521889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5143.874733] Lustre: lustre-OST0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 5145.572716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.579356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5145.595579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.621448] Lustre: server umount lustre-OST0000 complete [ 5155.749552] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c5cc8bf9800.ost_server_uuid 50 [ 5157.366483] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5171.178384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.179232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.195863] Lustre: Skipped 1 previous similar message [ 5171.216288] Lustre: Skipped 2 previous similar messages [ 5175.202552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5175.220365] Lustre: Skipped 1 previous similar message [ 5177.188653] Lustre: server umount lustre-MDT0000 complete [ 5180.384963] LustreError: 188983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.399148] LustreError: 188983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5180.674065] LustreError: 188962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545654 with bad export cookie 1951225855289687156 [ 5180.679853] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5180.696722] LustreError: 188962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5181.057762] Lustre: server umount lustre-MDT0001 complete [ 5190.322564] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:14:23 (1788545663) [ 5197.426355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.794333] LustreError: 192900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.904415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.508554] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5202.920630] LustreError: 192901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.047754] LustreError: 192900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.437234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.714684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5213.026139] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5219.207592] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5224.369963] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5228.961331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5229.206246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.046260] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5240.280509] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5243.589727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5245.976348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5246.152387] Lustre: lustre-OST0001: new disk, initializing [ 5246.156647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5246.161952] Lustre: 195743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5246.248565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.953493] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5252.212446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5252.228083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5252.392116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5261.441525] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5264.356792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5264.370400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.392118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5269.801120] Lustre: server umount lustre-OST0000 complete [ 5272.332544] LustreError: 194725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.975145] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5276.259562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5280.734155] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5280.961224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5288.423204] LustreError: 194723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.445966] LustreError: 194723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5298.660633] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.666946] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5298.667099] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5298.667104] Lustre: Skipped 4 previous similar messages [ 5298.675987] Lustre: Skipped 2 previous similar messages [ 5298.718517] LustreError: Skipped 1 previous similar message [ 5300.445058] Lustre: server umount lustre-OST0001 complete [ 5318.112276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5318.403583] Lustre: server umount lustre-MDT0000 complete [ 5322.239880] LustreError: 192882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545796 with bad export cookie 1951225855289688227 [ 5322.246166] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5322.254056] LustreError: 192882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5322.734399] Lustre: server umount lustre-MDT0001 complete [ 5347.481914] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:17:00 (1788545820) [ 5354.705166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.060853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5355.121274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5355.401729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.750979] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5368.789293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.879728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5368.905716] Lustre: 199180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5373.891852] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5381.253424] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5386.937848] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5393.554664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.686891] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5399.682559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5399.902474] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5406.483888] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5412.954811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.035189] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5420.139275] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5421.331370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5428.395268] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5431.783428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.790080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.800190] Lustre: Skipped 3 previous similar messages [ 5431.805390] Lustre: Skipped 9 previous similar messages [ 5438.011934] Lustre: server umount lustre-OST0000 complete [ 5439.246607] LustreError: 201029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.290858] LustreError: 201029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5457.382776] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.384464] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5457.407211] LustreError: Skipped 1 previous similar message [ 5458.803884] Lustre: server umount lustre-OST0001 complete [ 5467.988113] Lustre: server umount lustre-MDT0000 complete [ 5471.052794] LustreError: 199173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788545945 with bad export cookie 1951225855289689634 [ 5471.061283] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.069828] LustreError: 199173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5471.523697] Lustre: server umount lustre-MDT0001 complete [ 5493.177940] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:19:25 (1788545965) [ 5508.080220] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 5517.423326] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 5547.830988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.028631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5548.066838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5548.149396] Lustre: lustre-MDT0000: new disk, initializing [ 5548.241086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.250597] Lustre: Skipped 3 previous similar messages [ 5548.263487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5552.396296] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5561.348615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.427308] Lustre: 207275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5561.470331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5561.474375] Lustre: Skipped 1 previous similar message [ 5561.534568] Lustre: lustre-MDT0001: new disk, initializing [ 5561.655082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5561.663996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5565.207913] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5573.688861] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5579.353526] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5585.484651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.780329] Lustre: lustre-OST0000: new disk, initializing [ 5585.783290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5585.788947] Lustre: 209246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5591.089940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5591.105399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5591.240742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5591.416624] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5599.547618] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5604.072510] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5604.264507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5608.753540] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5608.943762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5612.002239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.021928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.025483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5612.039634] Lustre: Skipped 4 previous similar messages [ 5612.065875] Lustre: Skipped 9 previous similar messages [ 5625.312941] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5625.443269] Lustre: server umount lustre-OST0000 complete [ 5635.449963] Lustre: server umount lustre-MDT0000 complete [ 5637.603815] LustreError: 208343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.619357] LustreError: 208343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5639.087614] LustreError: 207266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546113 with bad export cookie 1951225855289691412 [ 5639.101192] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5639.472127] Lustre: server umount lustre-MDT0001 complete [ 5647.076407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.802625] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5659.983912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.087802] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5672.349489] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5678.272627] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5684.777228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5685.182840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.186914] Lustre: Skipped 4 previous similar messages [ 5691.542167] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5698.493643] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5703.658567] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5703.850811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5708.746244] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5709.043459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5712.865828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.872541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.877325] LustreError: Skipped 1 previous similar message [ 5712.886680] Lustre: Skipped 2 previous similar messages [ 5717.681480] Lustre: server umount lustre-OST0000 complete [ 5723.945608] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5728.673171] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5757.411437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.065876] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5771.356302] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.964377] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5777.262355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.273825] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.277568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5777.300715] Lustre: Skipped 1 previous similar message [ 5782.497621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5782.501769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5782.511862] LustreError: Skipped 1 previous similar message [ 5782.522303] Lustre: Skipped 10 previous similar messages [ 5794.272145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5794.510564] Lustre: server umount lustre-OST0000 complete [ 5805.057155] Lustre: server umount lustre-MDT0000 complete [ 5808.102759] LustreError: 211463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.141392] LustreError: 211463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5809.593801] LustreError: 216264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546283 with bad export cookie 1951225855289692301 [ 5809.597757] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5809.605527] LustreError: 216264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5810.126810] Lustre: server umount lustre-MDT0001 complete [ 5819.942985] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:24:52 (1788546292) [ 5826.707868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.811935] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5841.271612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.638177] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5856.431945] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5863.710299] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5871.440817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.551971] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5884.892995] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5888.647592] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.837218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5893.961491] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5894.170609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5900.430827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.621459] Lustre: lustre-OST0001: new disk, initializing [ 5900.624986] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5900.634956] Lustre: 220802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5906.906080] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5909.046162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5909.059478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5909.142933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5915.270422] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5918.979533] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5919.340856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5923.381516] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5923.591079] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5929.955984] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5929.960971] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.978588] LustreError: Skipped 2 previous similar messages [ 5930.014383] Lustre: Skipped 5 previous similar messages [ 5931.472355] Lustre: server umount lustre-OST0001 complete [ 5940.450837] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5940.893408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5946.367501] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5946.725511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5976.422366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.710359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5976.715102] Lustre: Skipped 5 previous similar messages [ 5982.196031] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5982.213244] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5982.227199] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5982.501680] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 5990.379512] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5991.848885] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6008.288894] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6008.359961] Lustre: server umount lustre-OST0001 complete [ 6032.756233] Lustre: server umount lustre-OST0000 complete [ 6038.512841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6038.529078] Lustre: Skipped 19 previous similar messages [ 6043.099046] Lustre: server umount lustre-MDT0000 complete [ 6046.766754] LustreError: 217411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546520 with bad export cookie 1951225855289693260 [ 6046.783059] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.796649] LustreError: 217411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6053.308336] Lustre: server umount lustre-MDT0001 complete [ 6075.925906] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:29:08 (1788546548) [ 6083.028462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.425382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6083.485113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6089.080319] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6097.767033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.863511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6097.888721] Lustre: 226050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6101.775744] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6109.021402] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6114.477503] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6121.783208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.994063] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6128.760314] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6136.194246] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6144.766257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6144.880589] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6151.159529] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6158.123736] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6159.642209] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6163.913521] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6164.098918] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6168.470610] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6168.701407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6174.239313] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c5cc5448800.ost_server_uuid 50 [ 6175.571893] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c5cc5448800.ost_server_uuid in IDLE state after 0 sec [ 6176.623603] Lustre: Permanently deactivating lustre-OST0001 [ 6176.639312] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6180.356317] Lustre: Permanently reactivating lustre-OST0001 [ 6186.469970] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.473948] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6186.499368] Lustre: Skipped 7 previous similar messages [ 6186.515163] LustreError: Skipped 3 previous similar messages [ 6190.395479] Lustre: server umount lustre-OST0001 complete [ 6191.591204] LustreError: 227892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.626389] LustreError: 227892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6201.268760] Lustre: server umount lustre-OST0000 complete [ 6219.232135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6219.654191] Lustre: server umount lustre-MDT0000 complete [ 6223.358550] LustreError: 226790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546697 with bad export cookie 1951225855289694555 [ 6223.363803] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6223.367735] LustreError: 226790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6223.622990] Lustre: server umount lustre-MDT0001 complete [ 6245.697115] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:31:58 (1788546718) [ 6252.572519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.829637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6252.873625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6252.884932] Lustre: Skipped 5 previous similar messages [ 6257.955337] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6265.620352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.704422] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6265.727922] Lustre: 232742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6270.640821] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6277.580967] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6283.282215] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6290.251300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6290.492936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6290.516297] Lustre: Permanently deactivating lustre-OST0000 [ 6290.532554] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6296.614924] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6297.519503] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6305.313694] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.980316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.113852] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6319.747744] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6319.856846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6328.250903] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.858446] Lustre: Permanently reactivating lustre-OST0000 [ 6329.883835] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6329.900845] Lustre: Skipped 2 previous similar messages [ 6336.589203] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6336.592351] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 000000007e0c011a, cur 1788546810 deadline 1788546800 last 1788546770 [ 6336.602597] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.649921] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6336.659153] Lustre: Skipped 1 previous similar message [ 6336.688301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6349.613350] Lustre: server umount lustre-OST0001 complete [ 6359.823529] Lustre: server umount lustre-OST0000 complete [ 6369.894894] Lustre: server umount lustre-MDT0000 complete [ 6373.526167] LustreError: 234610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788546847 with bad export cookie 1951225855289696207 [ 6373.540266] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.543475] LustreError: 234610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6374.037287] Lustre: server umount lustre-MDT0001 complete [ 6385.891871] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6389.407687] Key type lgssc unregistered [ 6389.716630] LNet: 237620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6389.721422] LNetError: 237620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6389.746849] LNet: Removed LNI 192.168.203.158@tcp [ 6390.836304] Key type .llcrypt unregistered [ 6390.838340] Key type ._llcrypt unregistered [ 6417.183654] Key type ._llcrypt registered [ 6417.189307] Key type .llcrypt registered [ 6417.342766] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6418.786063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.932584] alg: No test for adler32 (adler32-zlib) [ 6420.085450] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 6420.401950] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6422.160230] Key type lgssc registered [ 6423.285958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6434.076734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6434.087614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.543630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.499229] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6446.068512] LustreError: 239086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.100963] LustreError: 239086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6449.653537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.804581] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6449.843908] Lustre: 239076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6449.857432] Lustre: Permanently deactivating lustre-MDT0001 [ 6449.880349] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6450.211133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6453.141694] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6453.152859] LustreError: 239117:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6453.171433] LustreError: 239117:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6453.199842] LustreError: 239117:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 6454.430313] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6462.029825] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6468.721880] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6477.220085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.717565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.834846] LustreError: 240930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.856027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6487.028378] LustreError: 240930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.453604] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6492.140572] LustreError: 240931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.222091] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6497.860955] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6500.340043] LustreError: 241090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.357372] LustreError: 241090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6505.533755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.776547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6511.110586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6511.122194] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6513.327481] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6521.256832] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.499052] Lustre: Permanently reactivating lustre-MDT0001 [ 6525.515689] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6525.521428] Lustre: Skipped 2 previous similar messages [ 6528.261123] Lustre: lustre-MDT0001: Client efa5669e-8477-44dd-9301-c7d00ec88d09 (at 192.168.203.58@tcp) reconnecting [ 6535.220921] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6536.895895] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6543.042491] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6543.309212] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6548.886775] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6549.239306] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6552.024378] Lustre: Permanently deactivating lustre-MDT0001 [ 6552.039800] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6552.044958] Lustre: Skipped 2 previous similar messages [ 6559.416446] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6567.905655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6567.913053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.943248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.423767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.444888] Lustre: Skipped 1 previous similar message [ 6568.449943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.455539] Lustre: Skipped 3 previous similar messages [ 6572.532986] Lustre: server umount lustre-MDT0000 complete [ 6573.538669] LustreError: 239832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.572685] LustreError: 239832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6576.957052] LustreError: 239067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788547050 with bad export cookie 18379536669123471072 [ 6576.965736] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.970121] LustreError: 239067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6577.479552] Lustre: server umount lustre-MDT0001 complete [ 6583.986781] Lustre: server umount lustre-OST0000 complete [ 6597.388154] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:37:49 (1788547069) [ 6608.998304] Lustre: server umount lustre-OST0001 complete [ 6617.569198] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 6629.985359] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6666.470816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.698761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6666.705846] Lustre: Skipped 2 previous similar messages [ 6666.726798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6666.817810] Lustre: lustre-MDT0000: new disk, initializing [ 6666.879472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6666.897953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6671.992323] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6682.955445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.049801] Lustre: 247344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6683.104435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6683.107865] Lustre: Skipped 1 previous similar message [ 6683.193172] Lustre: lustre-MDT0001: new disk, initializing [ 6683.336207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6683.385153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6683.402161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6688.128214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6697.576764] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6703.626440] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6709.988229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.379874] Lustre: lustre-OST0000: new disk, initializing [ 6710.383940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6710.394497] Lustre: 249313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6710.525491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.498405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6716.511022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6716.676182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6717.544704] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6727.575864] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6729.519911] LustreError: 249838:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6738.502034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6738.742384] Lustre: lustre-OST0001: new disk, initializing [ 6738.746345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6738.751959] Lustre: 250427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6739.600332] LustreError: 249838:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6739.627680] LustreError: 247352:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6745.859714] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6747.223715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6747.233393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6747.291085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6749.704136] LustreError: 247352:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6756.146854] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6758.155849] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6762.283257] Lustre: lustre-OST0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 6762.986074] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6763.005513] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6763.032695] Lustre: Skipped 1 previous similar message [ 6766.369620] Lustre: server umount lustre-OST0001 complete [ 6767.367118] LustreError: 249299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.403890] LustreError: 249299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6788.582251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.587748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.595242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.595250] Lustre: Skipped 2 previous similar messages [ 6788.599900] LustreError: Skipped 2 previous similar messages [ 6788.655270] Lustre: Skipped 2 previous similar messages [ 6793.701350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.715207] Lustre: Skipped 2 previous similar messages [ 6798.304156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6798.642479] Lustre: server umount lustre-OST0000 complete [ 6803.948429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.952074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6803.974519] Lustre: Skipped 1 previous similar message [ 6803.995450] Lustre: Skipped 1 previous similar message [ 6808.956616] Lustre: server umount lustre-MDT0000 complete [ 6809.069751] LustreError: 248242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.087262] LustreError: 248242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6812.843223] LustreError: 247338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788547286 with bad export cookie 18379536669123474110 [ 6812.845675] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.858496] LustreError: 247338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6819.445581] Lustre: server umount lustre-MDT0001 complete [ 6831.685523] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 6835.253309] Key type lgssc unregistered [ 6835.672886] LNet: 252463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6835.680959] LNetError: 252463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6835.704981] LNet: Removed LNI 192.168.203.158@tcp [ 6836.700475] Key type .llcrypt unregistered [ 6836.703537] Key type ._llcrypt unregistered [ 6878.121594] Key type ._llcrypt registered [ 6878.124955] Key type .llcrypt registered [ 6878.249954] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 6879.375172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6879.400049] alg: No test for adler32 (adler32-zlib) [ 6880.755783] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 6881.100746] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 6882.776259] Key type lgssc registered [ 6884.107302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6892.664412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6892.681663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.144559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6894.211711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6894.402649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.137571] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6911.303689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.471119] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6911.539378] Lustre: 254666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6912.030232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6917.763670] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6926.354432] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6931.763278] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6938.143961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.355197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6938.645264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.796719] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 6948.112187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6953.662409] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7004.641299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7004.646305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.651415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7004.672821] Lustre: Skipped 1 previous similar message [ 7004.683479] Lustre: Skipped 1 previous similar message [ 7009.907745] Lustre: server umount lustre-OST0000 complete [ 7014.140897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7027.519901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7027.988225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.163199] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7034.184160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7034.188145] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7035.328523] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7043.394809] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.539810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7085.550090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7085.553323] LustreError: Skipped 2 previous similar messages [ 7085.560240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7085.614931] Lustre: Skipped 1 previous similar message [ 7090.116925] Lustre: server umount lustre-OST0000 complete [ 7095.778696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.780138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.786108] Lustre: Skipped 1 previous similar message [ 7095.789774] Lustre: Skipped 1 previous similar message [ 7100.901536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.908711] Lustre: Skipped 1 previous similar message [ 7101.701579] Lustre: server umount lustre-MDT0000 complete [ 7104.843735] LustreError: 254660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788547578 with bad export cookie 11754904296884852988 [ 7104.851353] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.858034] LustreError: 254660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7105.200123] Lustre: server umount lustre-MDT0001 complete [ 7115.993245] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7119.006804] Key type lgssc unregistered [ 7119.407631] LNet: 260090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7119.415023] LNetError: 260090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7119.443538] LNet: Removed LNI 192.168.203.158@tcp [ 7120.551092] Key type .llcrypt unregistered [ 7120.553082] Key type ._llcrypt unregistered [ 7144.826311] Key type ._llcrypt registered [ 7144.829532] Key type .llcrypt registered [ 7144.974416] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7146.098184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.169074] alg: No test for adler32 (adler32-zlib) [ 7147.233579] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 7147.457726] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7149.096168] Key type lgssc registered [ 7150.349100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7157.459034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7157.487029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.164352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.014994] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7169.510552] LustreError: 261505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.523483] LustreError: 261505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7172.495800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.083209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.975236] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7185.905918] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.972757] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.800321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.392070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.544102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7206.517788] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7213.901672] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.673135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7240.688382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.710204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7243.746032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7243.748511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7243.756557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.773029] Lustre: Skipped 1 previous similar message [ 7245.166335] Lustre: server umount lustre-OST0000 complete [ 7249.899319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7249.914577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.935226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7253.987425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7253.998477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.775426] Lustre: server umount lustre-MDT0000 complete [ 7259.105647] LustreError: 261506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.136435] LustreError: 261506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7259.451261] LustreError: 261487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788547733 with bad export cookie 15208226782535882042 [ 7259.459232] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.466837] LustreError: 261487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7259.788187] Lustre: server umount lustre-MDT0001 complete [ 7273.257305] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7276.737971] Key type lgssc unregistered [ 7277.012295] LNet: 265809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.016328] LNetError: 265809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.025528] LNet: Removed LNI 192.168.203.158@tcp [ 7277.723799] Key type .llcrypt unregistered [ 7277.725498] Key type ._llcrypt unregistered [ 7296.233237] Key type ._llcrypt registered [ 7296.236387] Key type .llcrypt registered [ 7296.353597] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7297.503348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.516754] alg: No test for adler32 (adler32-zlib) [ 7298.652380] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 7298.895831] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7300.544154] Key type lgssc registered [ 7301.754380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7308.999786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7309.015380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.418400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.670032] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7320.546802] LustreError: 267027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.575257] LustreError: 267027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7323.017882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.357705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7327.035172] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7333.257457] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7338.117967] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7343.189868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.555634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.228894] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7355.891921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7356.091896] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.118144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.119904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.120438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.128369] Lustre: Skipped 1 previous similar message [ 7366.146338] Lustre: Skipped 1 previous similar message [ 7371.234465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.241118] Lustre: Skipped 1 previous similar message [ 7372.138437] Lustre: server umount lustre-OST0000 complete [ 7376.354019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.367370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7376.386996] Lustre: Skipped 1 previous similar message [ 7376.406767] Lustre: Skipped 1 previous similar message [ 7381.475786] LustreError: 267028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.499880] LustreError: 267028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7381.596650] Lustre: server umount lustre-MDT0000 complete [ 7386.277238] LustreError: 267009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788547860 with bad export cookie 2839114682983150100 [ 7386.283671] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7386.322274] LustreError: 267009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7386.692660] Lustre: server umount lustre-MDT0001 complete [ 7399.880807] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7402.797812] Key type lgssc unregistered [ 7403.235108] LNet: 270703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7403.238973] LNetError: 270703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7403.250941] LNet: Removed LNI 192.168.203.158@tcp [ 7403.914794] Key type .llcrypt unregistered [ 7403.919851] Key type ._llcrypt unregistered [ 7425.941500] Key type ._llcrypt registered [ 7425.945497] Key type .llcrypt registered [ 7426.067985] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7427.283723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7427.396546] alg: No test for adler32 (adler32-zlib) [ 7428.479837] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 7428.806261] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7430.496190] Key type lgssc registered [ 7431.591184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7438.658579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7438.670631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.103867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.313942] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7450.614894] LustreError: 272115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.649719] LustreError: 272115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7454.048304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.443166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7459.702495] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7467.423664] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7472.734743] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7478.910639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.255912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.051427] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7485.427309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7492.243265] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7516.133762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7516.143944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.146623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.158275] Lustre: Skipped 1 previous similar message [ 7521.261262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.271558] Lustre: Skipped 2 previous similar messages [ 7521.808764] Lustre: server umount lustre-OST0000 complete [ 7525.858148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7525.874958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.891239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7531.475543] Lustre: server umount lustre-MDT0000 complete [ 7531.504366] LustreError: 272109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.518067] LustreError: 272109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7534.893201] LustreError: 272096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548008 with bad export cookie 11229092354278392837 [ 7534.895919] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7534.910878] LustreError: 272096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7535.272837] Lustre: server umount lustre-MDT0001 complete [ 7545.252687] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7547.587426] Key type lgssc unregistered [ 7547.827985] LNet: 276420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.836849] LNetError: 276420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.850758] LNet: Removed LNI 192.168.203.158@tcp [ 7548.677826] Key type .llcrypt unregistered [ 7548.679556] Key type ._llcrypt unregistered [ 7565.859862] Key type ._llcrypt registered [ 7565.862759] Key type .llcrypt registered [ 7565.989853] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7566.831789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7566.857818] alg: No test for adler32 (adler32-zlib) [ 7567.903514] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 7568.145582] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7569.808413] Key type lgssc registered [ 7570.945743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7578.530853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7578.551513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7580.075255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.519322] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7590.372089] LustreError: 277651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.393082] LustreError: 277651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7592.088100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.495164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.770586] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7603.550830] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7608.718617] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7615.210316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.525770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.914570] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7625.788828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7629.132925] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7641.065951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7641.067635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.073467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7641.105593] Lustre: Skipped 1 previous similar message [ 7642.486774] Lustre: server umount lustre-OST0000 complete [ 7646.177801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.181298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.192193] Lustre: Skipped 1 previous similar message [ 7646.197964] Lustre: Skipped 1 previous similar message [ 7651.297131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.302648] Lustre: Skipped 2 previous similar messages [ 7652.310556] Lustre: server umount lustre-MDT0000 complete [ 7655.598204] LustreError: 277618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548129 with bad export cookie 14339773624950224131 [ 7655.600456] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.606968] LustreError: 277618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7655.923882] Lustre: server umount lustre-MDT0001 complete [ 7666.234685] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7669.271462] Key type lgssc unregistered [ 7669.618910] LNet: 281351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7669.623919] LNetError: 281351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7669.637861] LNet: Removed LNI 192.168.203.158@tcp [ 7670.532389] Key type .llcrypt unregistered [ 7670.536269] Key type ._llcrypt unregistered [ 7682.476688] Key type ._llcrypt registered [ 7682.481157] Key type .llcrypt registered [ 7682.593703] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7699.233794] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 7714.509366] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 7715.833585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7715.863370] alg: No test for adler32 (adler32-zlib) [ 7716.983884] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 7717.227311] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 7718.888195] Key type lgssc registered [ 7719.947504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7752.227880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7752.277525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.502902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7753.540784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7753.604512] Lustre: lustre-MDT0000: new disk, initializing [ 7753.685459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.700990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7758.917710] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7769.935583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.102942] Lustre: 284898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7770.211914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7770.216569] Lustre: Skipped 1 previous similar message [ 7770.299074] Lustre: lustre-MDT0001: new disk, initializing [ 7770.425750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7770.506817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7770.532659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7775.206812] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7783.956065] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7789.305696] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7795.656851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.999772] Lustre: lustre-OST0000: new disk, initializing [ 7796.010610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7796.017557] Lustre: 286867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7796.119093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.522836] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7806.024162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7806.034719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7806.188225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7812.937506] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.872874] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7818.067341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7823.142838] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7823.393343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7826.913384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7826.917597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.930531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.941551] Lustre: Skipped 1 previous similar message [ 7826.984110] Lustre: Skipped 1 previous similar message [ 7831.710677] Lustre: server umount lustre-OST0000 complete [ 7837.154436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7837.163262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.172246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.188805] Lustre: Skipped 1 previous similar message [ 7837.205340] Lustre: Skipped 1 previous similar message [ 7841.955768] Lustre: server umount lustre-MDT0000 complete [ 7845.909397] LustreError: 284890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548319 with bad export cookie 5473121978981151606 [ 7845.912598] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.920708] LustreError: 284890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7846.478624] Lustre: server umount lustre-MDT0001 complete [ 7855.044345] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:58:47 (1788548327) [ 7861.732621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.388138] LustreError: 289280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7862.417841] LustreError: 289280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7862.497699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.599534] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7867.877789] LustreError: 289281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.001075] LustreError: 289280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.856686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.363757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7880.081892] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7887.278401] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7893.783229] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7901.244774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7901.661807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.269781] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 7914.566740] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7928.289404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7928.301838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.318123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.374753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.374789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.397065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.825678] Lustre: server umount lustre-OST0000 complete [ 7938.017228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7938.038733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7938.074767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7938.101755] Lustre: Skipped 1 previous similar message [ 7943.520782] Lustre: server umount lustre-MDT0000 complete [ 7946.721799] LustreError: 289280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.748604] LustreError: 289280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7946.778042] LustreError: 289260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548420 with bad export cookie 5473121978981152488 [ 7946.781833] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.800708] LustreError: 289260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.186390] Lustre: server umount lustre-MDT0001 complete [ 7957.914690] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 7960.972787] Key type lgssc unregistered [ 7961.267658] LNet: 292835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7961.273678] LNetError: 292835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7961.295340] LNet: Removed LNI 192.168.203.158@tcp [ 7962.083146] Key type .llcrypt unregistered [ 7962.087447] Key type ._llcrypt unregistered [ 7989.252135] Key type ._llcrypt registered [ 7989.254176] Key type .llcrypt registered [ 7989.344784] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8004.286954] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8005.610444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8005.826311] alg: No test for adler32 (adler32-zlib) [ 8006.965470] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 8007.319898] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8009.001040] Key type lgssc registered [ 8010.314438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.864446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.891118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.191547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8056.238951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8056.323803] Lustre: lustre-MDT0000: new disk, initializing [ 8056.453653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.473301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8061.411843] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8071.727613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.892749] Lustre: 296668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8071.957320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8071.964957] Lustre: Skipped 1 previous similar message [ 8072.052379] Lustre: lustre-MDT0001: new disk, initializing [ 8072.153108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8072.200439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8072.206554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8075.682214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8083.471420] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8087.807169] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8093.066853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8093.285852] Lustre: lustre-OST2710: new disk, initializing [ 8093.289720] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8093.295307] Lustre: 298637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8093.396424] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8099.425362] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8099.928542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8099.942081] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8100.021939] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8107.877323] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8113.493371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8113.642685] Lustre: lustre-OST03e8: new disk, initializing [ 8113.658866] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8113.667332] Lustre: 299701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.752693] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8119.342864] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8121.452041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8121.467225] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8121.560133] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8126.395539] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8133.537408] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8133.693394] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8137.559633] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8137.750087] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8141.701919] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8141.904134] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8146.183670] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8146.400673] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8152.550469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.551893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8152.560344] Lustre: Skipped 2 previous similar messages [ 8152.570259] Lustre: Skipped 3 previous similar messages [ 8156.237390] Lustre: server umount lustre-MDT0000 complete [ 8159.315122] LustreError: 296661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548633 with bad export cookie 814408026058603042 [ 8159.321039] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.327906] LustreError: 296661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8159.623906] Lustre: server umount lustre-MDT0001 complete [ 8173.156528] Lustre: server umount lustre-OST2710 complete [ 8187.020351] Lustre: server umount lustre-OST03e8 complete [ 8203.081585] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 8212.140297] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8243.930626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.112394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8244.140652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8244.205855] Lustre: lustre-MDT0000: new disk, initializing [ 8244.291723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.314709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8249.124752] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8259.388324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.495548] Lustre: 304952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8259.533680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8259.537463] Lustre: Skipped 1 previous similar message [ 8259.588902] Lustre: lustre-MDT0001: new disk, initializing [ 8259.654247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8259.690738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8259.703537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8263.974933] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8273.142105] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.403893] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8283.940766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8284.152368] Lustre: lustre-OST0000: new disk, initializing [ 8284.158636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8284.166296] Lustre: 306917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8284.255153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.197735] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8294.513554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8294.532447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8294.704258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8299.566378] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8305.036364] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8305.318350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8309.579973] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8309.840184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8315.362226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.371430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.385350] Lustre: Skipped 2 previous similar messages [ 8315.388310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.414797] Lustre: Skipped 1 previous similar message [ 8317.901045] Lustre: server umount lustre-OST0000 complete [ 8325.601791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.603720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.623253] Lustre: Skipped 1 previous similar message [ 8325.634372] Lustre: Skipped 1 previous similar message [ 8327.591168] Lustre: server umount lustre-MDT0000 complete [ 8330.728433] LustreError: 304958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.754145] LustreError: 304958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8332.132519] LustreError: 304943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548806 with bad export cookie 814408026058604722 [ 8332.139414] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.145460] LustreError: 304943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8332.596892] Lustre: server umount lustre-MDT0001 complete [ 8342.177187] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:06:54 (1788548814) [ 8344.266965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8346.321436] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:06:58 (1788548818) [ 8349.753511] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8373.727526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.009424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8374.036378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8374.244752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8378.583207] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8386.144712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.280836] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8386.330721] Lustre: 310774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8386.414462] LustreError: 310787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.726147] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8410.997423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.320515] Lustre: Denying initial registration attempt from nid 192.168.203.158@tcp, specified as failover [ 8411.333191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8411.353111] LustreError: 312230:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 8411.368678] LustreError: 312230:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8411.378710] LustreError: 312230:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8411.472253] Lustre: server umount lustre-OST0000 complete [ 8411.478210] LustreError: 312230:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8422.371360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8422.373970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.380203] LustreError: Skipped 1 previous similar message [ 8422.381204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8422.400927] Lustre: Skipped 1 previous similar message [ 8423.760818] Lustre: server umount lustre-MDT0000 complete [ 8427.489944] LustreError: 310787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.508406] LustreError: 310787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8427.704085] LustreError: 310768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788548901 with bad export cookie 814408026058605604 [ 8427.710389] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.723820] LustreError: 310768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8428.064464] Lustre: server umount lustre-MDT0001 complete [ 8438.063073] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8440.970392] Key type lgssc unregistered [ 8441.235118] LNet: 313351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8441.246124] LNetError: 313351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8441.259377] LNet: Removed LNI 192.168.203.158@tcp [ 8442.186403] Key type .llcrypt unregistered [ 8442.189202] Key type ._llcrypt unregistered [ 8452.650521] Key type ._llcrypt registered [ 8452.655859] Key type .llcrypt registered [ 8452.737855] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8453.519513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8453.556740] alg: No test for adler32 (adler32-zlib) [ 8454.717736] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 8454.922896] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8456.656523] Key type lgssc registered [ 8457.828533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8506.390204] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8514.683952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8514.712320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.995603] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8516.018921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8516.139957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.375732] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8527.976058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.126518] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8528.171970] Lustre: 316112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8528.423294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8533.043557] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8540.651194] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8546.530408] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.962735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.214466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8567.458491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8574.161595] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8581.949661] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.240148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8587.259184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.274084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8587.756260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8587.769164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.785466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8590.413772] Lustre: server umount lustre-OST0000 complete [ 8594.917085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8594.930280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.950216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8597.986757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8597.988258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.548471] Lustre: server umount lustre-MDT0000 complete [ 8605.269641] LustreError: 316104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549079 with bad export cookie 10445026140018997729 [ 8605.273806] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8605.280128] LustreError: 316104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8605.679096] Lustre: server umount lustre-MDT0001 complete [ 8616.335526] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8619.092075] Key type lgssc unregistered [ 8619.515459] LNet: 319680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8619.529587] LNetError: 319680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8619.563138] LNet: Removed LNI 192.168.203.158@tcp [ 8620.482205] Key type .llcrypt unregistered [ 8620.484674] Key type ._llcrypt unregistered [ 8649.091300] Key type ._llcrypt registered [ 8649.094810] Key type .llcrypt registered [ 8649.233533] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8650.898818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8651.063592] alg: No test for adler32 (adler32-zlib) [ 8652.169526] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 8652.440075] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8654.329592] Key type lgssc registered [ 8656.068793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8664.655532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8664.688410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.225499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.649653] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8676.840620] LustreError: 321095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.867354] LustreError: 321095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8681.092414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.418497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8687.105979] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8696.159532] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8702.254254] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8711.744322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8712.183392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.335685] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8725.246897] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8748.001846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8748.015989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.039994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8751.899387] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [ 8752.108505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.135889] Lustre: Skipped 1 previous similar message [ 8754.293156] Lustre: server umount lustre-MDT0000 complete [ 8757.001906] LustreError: 322938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8758.226027] LustreError: 321077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549232 with bad export cookie 2089381077584244039 [ 8758.234442] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8758.245041] LustreError: 321077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8762.116552] LustreError: 322938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.161232] LustreError: 322938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8762.174815] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [ 8762.195265] Lustre: Skipped 3 previous similar messages [ 8762.347269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.751809] Lustre: server umount lustre-MDT0001 complete [ 8771.548784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8780.288097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.840738] LustreError: 320404:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003d3dc4c0 x1875429688390656/t0(0) o250->MGC192.168.203.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8784.161376] LustreError: 322928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.182210] LustreError: 322928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8784.256877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.339148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8788.329935] LustreError: 324710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.361916] LustreError: 324710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8788.432642] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8793.459686] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8796.843982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.167840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8801.931358] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8802.283650] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8809.447602] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8811.101242] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8816.610473] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8817.943118] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8825.313197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8825.321438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.347585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8825.357736] Lustre: Skipped 2 previous similar messages [ 8827.883588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8829.373478] Lustre: server umount lustre-OST0000 complete [ 8838.113317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.128538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8838.129087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.129092] Lustre: Skipped 2 previous similar messages [ 8838.134794] Lustre: Skipped 2 previous similar messages [ 8839.602132] Lustre: server umount lustre-MDT0000 complete [ 8843.182138] LustreError: 321077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549317 with bad export cookie 2089381077584251767 [ 8843.188864] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8843.194416] LustreError: 321077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8843.902304] Lustre: server umount lustre-MDT0001 complete [ 8854.758613] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 8857.919209] Key type lgssc unregistered [ 8858.311718] LNet: 327383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8858.322758] LNetError: 327383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8858.355730] LNet: Removed LNI 192.168.203.158@tcp [ 8859.493067] Key type .llcrypt unregistered [ 8859.495588] Key type ._llcrypt unregistered [ 8885.409761] Key type ._llcrypt registered [ 8885.415189] Key type .llcrypt registered [ 8885.562269] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 8886.804795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8886.921055] alg: No test for adler32 (adler32-zlib) [ 8887.991630] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 8888.325743] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 8890.064209] Key type lgssc registered [ 8891.327944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8899.423099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8899.438221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.057071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.836816] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8910.307160] LustreError: 328797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.315648] LustreError: 328797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8914.105936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.515582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8919.582284] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8927.497145] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8933.656449] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8939.810155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.833353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8939.842865] Lustre: Skipped 1 previous similar message [ 8943.193730] Lustre: server umount lustre-MDT0000 complete [ 8945.120944] LustreError: 329773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8945.139660] LustreError: 329773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8947.027288] LustreError: 328778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549420 with bad export cookie 3629655141630218259 [ 8947.039270] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.044865] LustreError: 328778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8947.513221] Lustre: server umount lustre-MDT0001 complete [ 8962.863315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.168062] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8963.258626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8963.509349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.450700] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8976.810884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.912917] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8976.941867] Lustre: 331383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8977.297462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8982.344199] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 8989.467800] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8994.312391] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9003.452308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.712351] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9003.732173] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9003.741739] LustreError: 333269:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9003.749606] LustreError: 333269:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9003.760341] LustreError: 333269:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9003.833573] Lustre: server umount lustre-OST0000 complete [ 9003.837199] LustreError: 333269:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9012.910054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.111936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9013.268069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.002111] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9023.073398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9025.500857] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9032.543581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.679604] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9032.750628] Lustre: lustre-OST0001: new disk, initializing [ 9032.753857] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9032.759779] Lustre: 334811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9032.898046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9039.198899] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9039.488208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9039.499749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9039.629189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9047.077608] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9055.227292] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9055.228717] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.237373] LustreError: Skipped 1 previous similar message [ 9055.270090] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9055.275374] Lustre: Skipped 2 previous similar messages [ 9055.343661] Lustre: Skipped 1 previous similar message [ 9056.753373] Lustre: server umount lustre-OST0001 complete [ 9060.339830] LustreError: 333734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.375075] LustreError: 333734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9063.907691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9063.937533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9063.972363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9065.447683] LustreError: 335189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.469258] LustreError: 335189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9070.563740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9070.568458] Lustre: Skipped 3 previous similar messages [ 9070.578359] LustreError: 333734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9070.590537] LustreError: 333734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9075.685115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.701144] Lustre: Skipped 2 previous similar messages [ 9076.192131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9076.537313] Lustre: server umount lustre-OST0000 complete [ 9080.809058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.832836] Lustre: Skipped 1 previous similar message [ 9085.922157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.935468] Lustre: Skipped 3 previous similar messages [ 9086.930995] Lustre: server umount lustre-MDT0000 complete [ 9090.794798] LustreError: 331375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549564 with bad export cookie 3629655141630218812 [ 9090.797526] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9090.823171] LustreError: 331375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9091.325710] Lustre: server umount lustre-MDT0001 complete [ 9102.550909] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 9105.873450] Key type lgssc unregistered [ 9106.209642] LNet: 336770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9106.215133] LNetError: 336770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9106.234768] LNet: Removed LNI 192.168.203.158@tcp [ 9107.210603] Key type .llcrypt unregistered [ 9107.213894] Key type ._llcrypt unregistered [ 9177.671876] Key type ._llcrypt registered [ 9177.674517] Key type .llcrypt registered [ 9177.766958] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [ 9193.711666] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9194.887948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9195.007703] alg: No test for adler32 (adler32-zlib) [ 9196.009890] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 9196.285607] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 9197.972633] Key type lgssc registered [ 9199.012898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9230.847278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9230.871540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.115393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9232.137231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9232.222790] Lustre: lustre-MDT0000: new disk, initializing [ 9232.310940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9232.327869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9237.833727] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9250.252495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.396355] Lustre: 341699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9250.460204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9250.463865] Lustre: Skipped 1 previous similar message [ 9250.507775] Lustre: lustre-MDT0001: new disk, initializing [ 9250.578379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9250.628104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9250.638430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9255.937651] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9265.967548] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9272.016468] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9279.754594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9280.043330] Lustre: lustre-OST0000: new disk, initializing [ 9280.048323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9280.053892] Lustre: 343671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9280.149523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.954262] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9286.758876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9286.792807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9286.888636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9295.142433] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.364716] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9299.592644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9303.669162] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9303.880488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9307.617290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9307.624893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.636063] LustreError: Skipped 1 previous similar message [ 9307.643941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.658380] Lustre: Skipped 1 previous similar message [ 9307.677944] Lustre: Skipped 1 previous similar message [ 9311.974935] Lustre: server umount lustre-OST0000 complete [ 9317.345196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9317.352459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.366588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9322.978326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9322.988418] Lustre: Skipped 2 previous similar messages [ 9323.463675] Lustre: server umount lustre-MDT0000 complete [ 9328.062469] LustreError: 341693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549802 with bad export cookie 4201755702101600265 [ 9328.069235] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9328.076496] LustreError: 341693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9328.380377] Lustre: server umount lustre-MDT0001 complete [ 9337.326774] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:23:29 (1788549809) [ 9344.685848] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:23:37 (1788549817) [ 9353.461793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.889965] LustreError: 346428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.898715] LustreError: 346428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9353.934234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9359.001486] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9359.331113] LustreError: 346429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.428637] LustreError: 346428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.549327] LustreError: 346429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.580624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.824710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.197143] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9384.569426] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9390.280350] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9397.694577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9398.323077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.526995] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9412.733755] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9421.308906] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9424.084832] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9426.571262] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9428.499755] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9430.209217] Lustre: Failing over lustre-MDT0000 [ 9430.364399] Lustre: server umount lustre-MDT0000 complete [ 9430.818401] LustreError: 347525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9431.521523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9431.533133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9431.558249] Lustre: Skipped 1 previous similar message [ 9435.115048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.138681] Lustre: Skipped 1 previous similar message [ 9440.225568] LustreError: 347412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.243295] LustreError: 347412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9448.998715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.082698] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9449.302240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.375580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9451.271716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9453.391249] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9454.575948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9454.613968] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9454.655357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9462.274252] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9463.741108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9467.466504] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9469.498052] Lustre: Failing over lustre-MDT0000 [ 9469.743870] Lustre: server umount lustre-MDT0000 complete [ 9469.922439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9469.924349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.930894] LustreError: 346425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.930905] LustreError: 346425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9469.980165] Lustre: Skipped 2 previous similar messages [ 9489.571906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.834886] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.144239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.212771] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9493.781469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9495.075676] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9495.547744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9495.553236] Lustre: Skipped 2 previous similar messages [ 9495.566953] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9495.630618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9505.823291] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9507.359983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9516.002924] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.020473] Lustre: Skipped 1 previous similar message [ 9516.025549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9516.031673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9519.180623] Lustre: server umount lustre-MDT0000 complete [ 9521.126658] LustreError: 346429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.151245] LustreError: 346429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9523.295904] LustreError: 346408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788549997 with bad export cookie 4201755702101602974 [ 9523.308735] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9523.328045] LustreError: 346408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9523.828474] Lustre: server umount lustre-MDT0001 complete [ 9537.905489] Lustre: server umount lustre-OST0000 complete [ 9556.501829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.582415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.665175] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9571.848860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.824123] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9585.169646] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9591.002153] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9597.138481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9603.585817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9603.999463] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9611.348675] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9617.892219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9617.899129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.909969] Lustre: Skipped 1 previous similar message [ 9617.914959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9617.920072] Lustre: Skipped 2 previous similar messages [ 9618.157963] Lustre: server umount lustre-OST0000 complete [ 9629.153492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9629.164801] Lustre: Skipped 4 previous similar messages [ 9631.260328] Lustre: server umount lustre-MDT0000 complete [ 9633.253223] LustreError: 352765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.291106] LustreError: 352765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9634.960950] LustreError: 353500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550108 with bad export cookie 4201755702101603723 [ 9634.962116] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9634.968876] LustreError: 353500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9635.260662] Lustre: server umount lustre-MDT0001 complete [ 9645.505394] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [ 9648.259622] Key type lgssc unregistered [ 9648.512410] LNet: 356297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9648.527338] LNetError: 356297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9648.548358] LNet: Removed LNI 192.168.203.158@tcp [ 9649.310741] Key type .llcrypt unregistered [ 9649.312441] Key type ._llcrypt unregistered [ 9673.897871] Key type ._llcrypt registered [ 9673.901797] Key type .llcrypt registered [ 9674.025328] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [ 9674.917520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9675.051300] alg: No test for adler32 (adler32-zlib) [ 9676.139534] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [ 9676.346704] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [ 9678.008177] Key type lgssc registered [ 9679.266599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9687.403541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9687.425064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.900214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9693.693376] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9698.283713] LustreError: 357812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.303735] LustreError: 357812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9702.966075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.587513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.836478] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9716.804794] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9722.537479] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9729.568574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9729.920990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9734.007404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9736.346329] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9743.858524] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9758.694775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9758.696140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9758.734206] Lustre: Skipped 1 previous similar message [ 9758.762219] Lustre: Skipped 2 previous similar messages [ 9760.576702] Lustre: server umount lustre-MDT0000 complete [ 9763.811994] LustreError: 357812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.857321] LustreError: 357812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9764.516668] LustreError: 357793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550238 with bad export cookie 14924136759196972990 [ 9764.527690] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9764.538363] LustreError: 357793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9765.142354] Lustre: server umount lustre-MDT0001 complete [ 9779.675358] Lustre: server umount lustre-OST0000 complete [ 9794.465517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9795.043068] LustreError: 361540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.166151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9800.165244] LustreError: 361541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.449505] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9805.284000] LustreError: 361540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.365712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.813055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.540579] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9824.879881] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9830.912315] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9838.031398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9838.381684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9840.433836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9845.052108] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9852.237107] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9865.188086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9865.193991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9865.202528] Lustre: Skipped 3 previous similar messages [ 9865.216170] Lustre: Skipped 2 previous similar messages [ 9869.684865] Lustre: server umount lustre-MDT0000 complete [ 9870.319893] LustreError: 361536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9870.344538] LustreError: 361536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9873.298331] LustreError: 361521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550347 with bad export cookie 14924136759196974103 [ 9873.306814] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9873.709925] Lustre: server umount lustre-MDT0001 complete [ 9888.258638] Lustre: server umount lustre-OST0000 complete [ 9905.559474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.179952] LustreError: 365451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.299632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.983728] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9921.343770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.091626] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9933.636754] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.677987] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9948.298851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9948.761306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9948.770782] Lustre: Skipped 1 previous similar message [ 9951.887784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9955.469941] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [ 9964.398332] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9971.681539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.698546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9971.699039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9971.701853] Lustre: Skipped 1 previous similar message [ 9971.724342] Lustre: Skipped 1 previous similar message [ 9975.441625] Lustre: server umount lustre-OST0000 complete [ 9981.921759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9981.923684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9981.935407] Lustre: Skipped 1 previous similar message [ 9987.043954] LustreError: 367895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.071814] LustreError: 367895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9988.013843] Lustre: server umount lustre-MDT0000 complete [ 9991.377202] LustreError: 365434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550465 with bad export cookie 14924136759196975125 [ 9991.380426] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9991.392193] LustreError: 365434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9991.651617] Lustre: server umount lustre-MDT0001 complete [10004.903610] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [10008.680232] Key type lgssc unregistered [10009.030993] LNet: 368984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10009.044255] LNetError: 368984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10009.064683] LNet: Removed LNI 192.168.203.158@tcp [10010.221079] Key type .llcrypt unregistered [10010.223270] Key type ._llcrypt unregistered [10038.467944] Key type ._llcrypt registered [10038.473809] Key type .llcrypt registered [10038.606264] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10039.744205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10039.787093] alg: No test for adler32 (adler32-zlib) [10041.042990] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [10041.352831] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [10043.113110] Key type lgssc registered [10044.476369] Lustre: Echo OBD driver; http://www.lustre.org/ [10053.045055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10053.077245] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.096209] LustreError: 370422:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10053.129411] LustreError: 370422:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10053.135727] LustreError: 370422:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10053.145419] LustreError: 370422:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10053.163859] LustreError: 370422:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10061.519312] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10061.535438] LustreError: 370820:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10061.589625] LustreError: 370820:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10061.602884] LustreError: 370820:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10061.615707] LustreError: 370820:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10061.632375] LustreError: 370820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10081.109384] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [10084.086067] Key type lgssc unregistered [10084.418164] LNet: 371823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10084.426671] LNetError: 371823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10084.452341] LNet: Removed LNI 192.168.203.158@tcp [10085.302784] Key type .llcrypt unregistered [10085.306155] Key type ._llcrypt unregistered [10104.376197] Key type ._llcrypt registered [10104.377828] Key type .llcrypt registered [10104.553860] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [10122.321069] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10123.291212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10123.396370] alg: No test for adler32 (adler32-zlib) [10124.515411] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [10124.783636] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [10126.560178] Key type lgssc registered [10127.735140] Lustre: Echo OBD driver; http://www.lustre.org/ [10163.852156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10163.899873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10165.268729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10165.292741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10165.374117] Lustre: lustre-MDT0000: new disk, initializing [10165.503494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10165.517522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10170.100785] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10180.116555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.249496] Lustre: 374978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10180.329076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10180.336270] Lustre: Skipped 1 previous similar message [10180.427453] Lustre: lustre-MDT0001: new disk, initializing [10180.591738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10180.656451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10180.672625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10184.710203] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10194.537512] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10200.690804] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10208.151218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10208.497255] Lustre: lustre-OST0000: new disk, initializing [10208.501742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10208.512535] Lustre: 376948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10208.663854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10215.013563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10215.028265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10215.218341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10215.722937] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10224.639397] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10229.675900] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10229.893872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10234.957129] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10235.153034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10241.000970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10241.004490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.009558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10241.015746] LustreError: Skipped 1 previous similar message [10241.052513] Lustre: Skipped 1 previous similar message [10243.347839] Lustre: server umount lustre-OST0000 complete [10247.137272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10247.146431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.166645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.177259] Lustre: Skipped 1 previous similar message [10251.233844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.239950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.262696] Lustre: Skipped 1 previous similar message [10253.292609] Lustre: server umount lustre-MDT0000 complete [10256.947754] LustreError: 374969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550730 with bad export cookie 13923059194063069944 [10256.956111] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.969067] LustreError: 374969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10257.407680] Lustre: server umount lustre-MDT0001 complete [10265.840954] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10267.693772] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:39:00 (1788550740) [10275.105590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.853386] LustreError: 379407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10275.876991] LustreError: 379407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10275.950404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.442156] LustreError: 379408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.422408] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10285.556994] LustreError: 379407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.659447] LustreError: 379408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.986177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.439121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10296.530487] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10304.101393] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10310.369586] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10318.083132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.447398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.862520] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10334.131906] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10342.041371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.297739] Lustre: lustre-OST0001: new disk, initializing [10342.306428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10342.315510] Lustre: 382247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10342.436425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10348.764350] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10350.695098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10350.705563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10350.769898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10357.792528] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10364.706418] Lustre: lustre-OST0001: Not available for connect from 192.168.203.58@tcp (stopping) [10366.435981] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10366.454074] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10367.799673] Lustre: server umount lustre-OST0001 complete [10369.795554] LustreError: 381359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10381.817798] LustreError: 381231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10381.846073] LustreError: 381231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10386.914756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.920453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10386.935506] Lustre: Skipped 2 previous similar messages [10386.946365] Lustre: Skipped 3 previous similar messages [10390.776930] Lustre: server umount lustre-OST0000 complete [10395.105468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10395.112251] LustreError: Skipped 1 previous similar message [10395.117674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10395.134220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10400.496530] Lustre: server umount lustre-MDT0000 complete [10402.276962] LustreError: 379407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.309031] LustreError: 379407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10404.891090] LustreError: 379388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788550878 with bad export cookie 13923059194063070826 [10404.903666] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10404.916336] LustreError: 379388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10411.329461] Lustre: server umount lustre-MDT0001 complete [10423.629422] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [10426.708509] Key type lgssc unregistered [10426.961326] LNet: 384230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10426.970234] LNetError: 384230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10426.983948] LNet: Removed LNI 192.168.203.158@tcp [10428.109435] Key type .llcrypt unregistered [10428.111932] Key type ._llcrypt unregistered [10482.836355] Key type ._llcrypt registered [10482.846095] Key type .llcrypt registered [10482.981822] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10484.288742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10484.333165] alg: No test for adler32 (adler32-zlib) [10485.383597] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [10485.659398] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [10487.409057] Key type lgssc registered [10488.842700] Lustre: Echo OBD driver; http://www.lustre.org/ [10496.881255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10496.893165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.123537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10498.157442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10498.331703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10502.305848] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10509.771900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.936235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10510.003968] Lustre: 387028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10510.340921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10514.833667] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10521.991561] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10527.200993] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.790727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10530.801137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.813594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10535.377238] Lustre: server umount lustre-MDT0000 complete [10539.247645] LustreError: 387019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551013 with bad export cookie 11910271308588847628 [10539.257897] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10539.269932] LustreError: 387019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10539.588175] Lustre: server umount lustre-MDT0001 complete [10548.677722] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:43:41 (1788551021) [10554.669336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.329304] LustreError: 389539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10555.347712] LustreError: 389539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10555.370578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10560.483215] LustreError: 389540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.620782] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10565.603150] LustreError: 389539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.737975] LustreError: 389540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.615143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.180883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10577.145732] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10583.644607] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10589.442615] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10595.239357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10595.441321] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10595.632152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10600.519157] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10607.692644] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10615.867352] Lustre: Permanently deactivating lustre-OST0000 [10615.899204] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10625.039492] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10626.993272] LustreError: 392575:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10627.001580] LustreError: 392575:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10627.011229] LustreError: 392575:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10630.072836] LustreError: 392624:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10630.087614] LustreError: 392624:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10638.367583] Lustre: server umount lustre-OST0000 complete [10641.230688] LustreError: 392873:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10641.243231] LustreError: 392873:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10643.937894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.955931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.977505] Lustre: Skipped 1 previous similar message [10643.985451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10643.996309] Lustre: Skipped 2 previous similar messages [10648.545649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10648.555884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.665277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10653.669123] Lustre: Skipped 2 previous similar messages [10654.850461] Lustre: server umount lustre-MDT0000 complete [10658.255037] LustreError: 389519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551132 with bad export cookie 11910271308588848216 [10658.256223] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10658.268917] LustreError: 389519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10668.641948] Lustre: server umount lustre-MDT0001 complete [10677.363120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.210447] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10684.695947] LustreError: 394032:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10690.164640] Lustre: 394370:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.158@tcp [10692.099926] Lustre: server umount MGS complete [10702.344382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10702.763482] LustreError: 395095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10702.782332] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10702.795254] Lustre: Skipped 1 previous similar message [10702.919497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10707.497337] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10713.060991] LustreError: 395096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10713.076129] LustreError: 395096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10716.191614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.515122] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10720.901404] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10728.395642] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10733.929141] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10735.613368] Lustre: Permanently reactivating lustre-OST0000 [10735.660366] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10735.669648] Lustre: Skipped 2 previous similar messages [10762.720571] LustreError: 396877:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10769.728818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.062919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10770.074389] Lustre: Skipped 1 previous similar message [10775.713724] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10782.337395] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10787.809749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10787.821374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10787.825456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10787.838112] Lustre: Skipped 1 previous similar message [10793.590656] Lustre: server umount lustre-OST0000 complete [10798.051321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.056658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10798.075728] Lustre: Skipped 2 previous similar messages [10798.096355] Lustre: Skipped 3 previous similar messages [10811.360134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10811.507530] Lustre: server umount lustre-MDT0000 complete [10813.409960] LustreError: 396131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.441449] LustreError: 396131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10814.953360] LustreError: 395077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551288 with bad export cookie 11910271308588849266 [10814.958995] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10814.967911] LustreError: 395077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10815.290097] Lustre: server umount lustre-MDT0001 complete [10824.038766] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [10827.048691] Key type lgssc unregistered [10827.372765] LNet: 399461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10827.390416] LNetError: 399461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10827.416816] LNet: Removed LNI 192.168.203.158@tcp [10828.257274] Key type .llcrypt unregistered [10828.259114] Key type ._llcrypt unregistered [10847.553185] Key type ._llcrypt registered [10847.555942] Key type .llcrypt registered [10847.709610] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [10863.693309] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [10865.343255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10865.389276] alg: No test for adler32 (adler32-zlib) [10866.543941] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [10866.942152] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [10868.672257] Key type lgssc registered [10869.775992] Lustre: Echo OBD driver; http://www.lustre.org/ [10901.719327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10901.754743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.983603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10903.055686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10903.146513] Lustre: lustre-MDT0000: new disk, initializing [10903.251324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10903.274603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10907.702824] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10917.406787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10917.545524] Lustre: 402614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10917.600938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10917.605216] Lustre: Skipped 1 previous similar message [10917.670551] Lustre: lustre-MDT0001: new disk, initializing [10917.780530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10917.827078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10917.840447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10922.339586] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10931.016266] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10936.089758] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10941.577034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10941.793179] Lustre: lustre-OST0000: new disk, initializing [10941.796603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10941.801927] Lustre: 404583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10941.878379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10947.731824] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [10948.681065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10948.687584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10948.763213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10955.519730] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10958.873539] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10959.011072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10962.127110] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10962.308325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10964.454529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10964.455560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10964.461315] LustreError: Skipped 1 previous similar message [10964.463557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10964.483794] Lustre: Skipped 1 previous similar message [10969.574115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10969.576970] Lustre: Skipped 1 previous similar message [10969.774246] Lustre: server umount lustre-OST0000 complete [10974.177188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10974.195058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10974.210277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10979.664355] Lustre: server umount lustre-MDT0000 complete [10982.970058] LustreError: 402609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551456 with bad export cookie 6303349676825649765 [10982.972381] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10982.980035] LustreError: 402609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10983.347327] Lustre: server umount lustre-MDT0001 complete [10990.603109] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10992.224641] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:51:04 (1788551464) [10998.090342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.448843] LustreError: 407044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.462627] LustreError: 407044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10998.505288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.173362] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11003.874342] LustreError: 407045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.998251] LustreError: 407044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.263553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.500723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11013.330287] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11019.891382] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.436694] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11032.280944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.663982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11038.480772] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11045.261502] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11046.899837] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11049.157595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11054.569802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11054.570157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11054.582665] Lustre: Skipped 1 previous similar message [11054.594887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11054.612074] Lustre: Skipped 2 previous similar messages [11059.195349] Lustre: server umount lustre-OST0000 complete [11064.805777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11064.807782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.824435] Lustre: Skipped 1 previous similar message [11068.691349] Lustre: server umount lustre-MDT0000 complete [11069.922821] LustreError: 407040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.952284] LustreError: 407040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11072.238187] LustreError: 407026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551546 with bad export cookie 6303349676825650647 [11072.247930] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11072.251041] LustreError: 407026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11072.472142] Lustre: server umount lustre-MDT0001 complete [11082.204500] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11084.794420] Key type lgssc unregistered [11085.053437] LNet: 410769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11085.059994] LNetError: 410769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11085.072204] LNet: Removed LNI 192.168.203.158@tcp [11085.872494] Key type .llcrypt unregistered [11085.875719] Key type ._llcrypt unregistered [11106.036612] Key type ._llcrypt registered [11106.041459] Key type .llcrypt registered [11106.116168] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11106.396286] Key type .llcrypt unregistered [11106.398239] Key type ._llcrypt unregistered [11123.376545] Key type ._llcrypt registered [11123.380776] Key type .llcrypt registered [11123.474603] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11124.121261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11124.136513] alg: No test for adler32 (adler32-zlib) [11125.105379] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11125.314177] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11126.992208] Key type lgssc registered [11127.911763] Lustre: Echo OBD driver; http://www.lustre.org/ [11134.365870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11134.396807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.726492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.347871] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11145.191640] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.217995] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11145.517925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11145.752019] LustreError: 412971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.777557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11150.822717] LustreError: 412976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.842718] LustreError: 412976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11151.039876] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11155.938313] LustreError: 412971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11155.951779] LustreError: 412971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11157.434932] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11161.057589] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.087366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11161.089665] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11162.811142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11163.122942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11166.874176] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11173.351967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11173.363635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.364242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11178.476161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11178.484091] Lustre: Skipped 1 previous similar message [11179.293374] Lustre: server umount lustre-OST0000 complete [11183.586425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11183.586587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11183.589524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11183.589532] Lustre: Skipped 1 previous similar message [11183.595180] Lustre: Skipped 2 previous similar messages [11188.582127] Lustre: server umount lustre-MDT0000 complete [11188.711141] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.733249] LustreError: 412977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11191.724910] LustreError: 412957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551665 with bad export cookie 13043212645860037083 [11191.727238] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.740178] LustreError: 412957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11192.052991] Lustre: server umount lustre-MDT0001 complete [11201.198709] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11203.643597] Key type lgssc unregistered [11203.923612] LNet: 416122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11203.928971] LNetError: 416122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11203.950639] LNet: Removed LNI 192.168.203.158@tcp [11204.733864] Key type .llcrypt unregistered [11204.736495] Key type ._llcrypt unregistered [11225.072952] Key type ._llcrypt registered [11225.075272] Key type .llcrypt registered [11225.159634] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11226.107838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11226.192154] alg: No test for adler32 (adler32-zlib) [11227.236970] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11227.494326] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11229.208184] Key type lgssc registered [11230.158190] Lustre: Echo OBD driver; http://www.lustre.org/ [11237.190240] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11237.224886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11252.832376] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11257.952314] LustreError: 417488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.158@tcp: failed processing log, type 4: rc = -110 [11284.577724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11290.123547] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11295.958846] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.839706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.340945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.410096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11306.139151] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11307.496722] LustreError: 417514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.510348] LustreError: 417514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11312.617939] LustreError: 417513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.216524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.565447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11316.935219] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11322.878748] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.509360] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11334.114842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11334.121600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.135984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11335.138862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11335.142127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11339.978705] Lustre: server umount lustre-OST0000 complete [11344.355723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11344.377436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.394944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11344.398358] Lustre: Skipped 1 previous similar message [11349.100288] Lustre: server umount lustre-MDT0000 complete [11350.498931] LustreError: 418527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11350.515899] LustreError: 418527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11352.029838] LustreError: 417496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551825 with bad export cookie 7563840857121384645 [11352.036915] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.049025] LustreError: 417496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11352.348537] Lustre: server umount lustre-MDT0001 complete [11361.354983] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11363.586423] Key type lgssc unregistered [11363.853688] LNet: 421054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11363.865128] LNetError: 421054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11363.883528] LNet: Removed LNI 192.168.203.158@tcp [11364.457562] Key type .llcrypt unregistered [11364.460210] Key type ._llcrypt unregistered [11382.472763] Key type ._llcrypt registered [11382.478127] Key type .llcrypt registered [11382.578695] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11383.242508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11383.252262] alg: No test for adler32 (adler32-zlib) [11384.234503] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11384.403121] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11386.034812] Key type lgssc registered [11386.912818] Lustre: Echo OBD driver; http://www.lustre.org/ [11392.883839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11392.906702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.312738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11398.093643] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11404.774755] LustreError: 422446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.786695] LustreError: 422446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11405.306569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.572059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11409.221251] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11415.013640] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11419.686833] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11425.744774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11426.128979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11430.610643] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11435.714762] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11439.468856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11444.710474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.713815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11444.726221] Lustre: Skipped 2 previous similar messages [11444.741927] Lustre: Skipped 2 previous similar messages [11447.574405] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.58@tcp (stopping) [11449.827566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.843415] Lustre: Skipped 2 previous similar messages [11450.328608] Lustre: server umount lustre-MDT0000 complete [11452.432599] LustreError: 422427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.58@tcp arrived at 1788551926 with bad export cookie 1642573560412296662 [11452.460800] LustreError: 422427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11454.156747] LustreError: 422426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788551928 with bad export cookie 1642573560412295997 [11454.168210] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11454.947122] LustreError: 424284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.954241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.972979] LustreError: 424284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11455.000801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11460.065853] LustreError: 422447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.066483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11460.087158] LustreError: 422447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11460.393923] Lustre: server umount lustre-OST0000 complete [11464.165869] LustreError: 422446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.176431] LustreError: 422446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11465.559896] Lustre: server umount lustre-MDT0001 complete [11473.570991] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11476.142213] Key type lgssc unregistered [11476.364260] LNet: 426129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11476.378120] LNetError: 426129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11476.393198] LNet: Removed LNI 192.168.203.158@tcp [11477.274709] Key type .llcrypt unregistered [11477.281171] Key type ._llcrypt unregistered [11499.985303] Key type ._llcrypt registered [11499.992195] Key type .llcrypt registered [11500.155753] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11501.717321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11501.798758] alg: No test for adler32 (adler32-zlib) [11503.216723] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11503.697959] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11505.496159] Key type lgssc registered [11507.057594] Lustre: Echo OBD driver; http://www.lustre.org/ [11514.609728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11514.622450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.879738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11520.446952] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11525.096727] LustreError: 427542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.119169] LustreError: 427542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11528.337054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.716377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11532.442330] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11539.203432] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11543.452767] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11549.593759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.952833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11556.913753] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11560.260607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11563.818657] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11569.636710] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11569.644644] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11569.663693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11570.489040] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [11570.675829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11573.404990] Lustre: server umount lustre-MDT0001 complete [11575.778455] LustreError: 427543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.801376] LustreError: 427543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11580.902795] LustreError: 429839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.903859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11580.932594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11580.942480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11580.946630] Lustre: Skipped 2 previous similar messages [11584.448887] Lustre: server umount lustre-OST0000 complete [11586.024989] LustreError: 427543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.022098] Lustre: server umount lustre-MDT0000 complete [11600.867546] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11603.702696] Key type lgssc unregistered [11604.000610] LNet: 431238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11604.012860] LNetError: 431238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11604.039823] LNet: Removed LNI 192.168.203.158@tcp [11604.906793] Key type .llcrypt unregistered [11604.909130] Key type ._llcrypt unregistered [11625.638104] Key type ._llcrypt registered [11625.645334] Key type .llcrypt registered [11625.725782] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [11637.596041] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11638.326843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11638.373272] alg: No test for adler32 (adler32-zlib) [11639.425690] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11639.624029] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11641.257274] Key type lgssc registered [11642.320957] Lustre: Echo OBD driver; http://www.lustre.org/ [11674.745400] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11690.291820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.316179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.664432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11691.714829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11691.778068] Lustre: lustre-MDT0000: new disk, initializing [11691.856635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11691.875356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11696.949053] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11712.112661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.370340] Lustre: lustre-OST0000: new disk, initializing [11712.379541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11712.387436] Lustre: Skipped 1 previous similar message [11712.396567] Lustre: 436249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11712.477035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11717.703915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11717.717410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11717.745712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11718.327808] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11727.105515] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11743.984546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.248436] Lustre: 435229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11744.341836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11744.404142] Lustre: lustre-MDT0001: new disk, initializing [11744.531802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11744.580324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11744.608561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11749.201522] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11756.887948] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11757.055259] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11761.479075] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11761.734525] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11766.034615] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [11767.267249] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11767.283274] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.303596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11767.314329] Lustre: Skipped 1 previous similar message [11771.148034] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.58@tcp (stopping) [11772.441747] Lustre: server umount lustre-MDT0001 complete [11777.515521] LustreError: 435236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.535232] LustreError: 435236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11782.646404] LustreError: 435234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.676337] LustreError: 435234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11787.748266] LustreError: 437573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.776764] LustreError: 437573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11792.866821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11792.868148] LustreError: 437958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.886833] Lustre: Skipped 1 previous similar message [11792.925872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11797.985260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11797.985694] LustreError: 435240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.128571] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11804.272080] Lustre: server umount lustre-OST0000 complete [11814.300473] Lustre: server umount lustre-MDT0000 complete [11829.192567] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11831.902695] Key type lgssc unregistered [11832.188774] LNet: 439999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11832.198523] LNetError: 439999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11832.222531] LNet: Removed LNI 192.168.203.158@tcp [11832.943950] Key type .llcrypt unregistered [11832.946179] Key type ._llcrypt unregistered [11871.377490] Key type ._llcrypt registered [11871.379540] Key type .llcrypt registered [11871.449953] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [11871.723460] Key type .llcrypt unregistered [11871.727228] Key type ._llcrypt unregistered [11886.609158] Key type ._llcrypt registered [11886.611749] Key type .llcrypt registered [11886.714833] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [11887.890816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11887.930046] alg: No test for adler32 (adler32-zlib) [11888.994960] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [11889.244487] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [11890.976267] Key type lgssc registered [11892.217982] Lustre: Echo OBD driver; http://www.lustre.org/ [11915.662512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11915.692434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.037449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11917.133731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11917.319285] Lustre: lustre-MDT0000: new disk, initializing [11917.576085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11917.636935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11922.490912] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11932.872862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.944531] Lustre: 443366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11932.986399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11932.988681] Lustre: Skipped 1 previous similar message [11933.038870] Lustre: lustre-MDT0001: new disk, initializing [11933.102288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11933.139602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11933.154094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11936.655355] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11944.674445] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11948.874742] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11954.388991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11954.726807] Lustre: lustre-OST0000: new disk, initializing [11954.729527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11954.736531] Lustre: 445332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11954.810578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11960.388209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11960.396988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11960.516136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11960.570728] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [11969.502407] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11979.237726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11979.245216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.263439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.283244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.284551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11981.304880] Lustre: Skipped 1 previous similar message [11982.904459] Lustre: server umount lustre-MDT0000 complete [11986.064756] LustreError: 443358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788552460 with bad export cookie 907933435712614245 [11986.071470] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11986.332488] Lustre: server umount lustre-MDT0001 complete [12000.110576] Lustre: server umount lustre-OST0000 complete [12009.289165] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:08:01 (1788552481) [12016.128308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.542419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12020.193265] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12021.735017] LustreError: 447361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.756309] LustreError: 447361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12026.849181] LustreError: 447360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.520326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.964438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12033.257602] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12040.234638] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12045.567752] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12064.634160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12064.875218] Lustre: Found index 0 for lustre-OST0000, updating log [12065.110059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12071.300063] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12079.111953] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12087.672063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12098.017228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12098.018648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.032063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12098.032071] Lustre: Skipped 2 previous similar messages [12098.055754] Lustre: Skipped 1 previous similar message [12105.187241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12105.198235] Lustre: Skipped 3 previous similar messages [12109.280124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12109.560268] Lustre: server umount lustre-OST0000 complete [12126.177139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12126.185256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12126.201992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12126.208859] Lustre: Skipped 1 previous similar message [12127.842478] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [12132.535196] Lustre: server umount lustre-MDT0000 complete [12136.584334] LustreError: 447340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788552610 with bad export cookie 907933435712615638 [12136.586620] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12136.602757] LustreError: 447340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12137.068715] Lustre: server umount lustre-MDT0001 complete [12146.535670] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:10:18 (1788552618) [12764.894356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12765.703922] LustreError: 453791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.819834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12770.734109] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12771.306438] LustreError: 453792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12775.396418] LustreError: 453791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12780.180505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12780.640610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12785.231659] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12791.823545] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12797.326402] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12816.874773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12817.152555] Lustre: Found index 0 for lustre-OST0000, updating log [12817.413540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12818.017442] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [12823.750452] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [12830.383318] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12832.494099] Lustre: server umount lustre-OST0000 complete [12836.833137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12836.836820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12836.844047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12836.844054] Lustre: Skipped 3 previous similar messages [12836.870749] Lustre: Skipped 2 previous similar messages [12837.922820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12837.927073] Lustre: Skipped 2 previous similar messages [12839.976876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12839.983139] Lustre: Skipped 3 previous similar messages [12844.065847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12844.068602] Lustre: Skipped 6 previous similar messages [12850.144128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12850.277612] Lustre: server umount lustre-MDT0000 complete [12851.233540] LustreError: 453792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.247269] LustreError: 453792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12853.898674] LustreError: 453771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788553327 with bad export cookie 907933435712616723 [12853.902889] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12853.914674] LustreError: 453771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12854.205489] Lustre: server umount lustre-MDT0001 complete [12877.409195] LNet: 1 local NIs in recovery (showing 1): 192.168.203.158@tcp [12883.417272] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [12892.352716] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [12930.479234] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:23:22 (1788553402) [12945.842886] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [12948.072352] Key type lgssc unregistered [12948.312516] LNet: 460406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12948.319720] LNetError: 460406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13068.135076] LNet: Removed LNI 192.168.203.158@tcp [13069.174117] Key type .llcrypt unregistered [13069.177604] Key type ._llcrypt unregistered [13087.259719] Key type ._llcrypt registered [13087.262276] Key type .llcrypt registered [13087.342074] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13088.404228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13088.416713] alg: No test for adler32 (adler32-zlib) [13089.425549] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [13089.612799] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [13091.256164] Key type lgssc registered [13092.250079] Lustre: Echo OBD driver; http://www.lustre.org/ [13096.120049] LNet: Added LNI 192.168.203.158@tcp100 [8/256/0/180] [13096.995574] LNet: Added LNI 192.168.203.158@tcp101 [8/256/0/180] [13097.784775] LNet: Added LNI 192.168.203.158@tcp102 [8/256/0/180] [13098.726831] LNet: Added LNI 192.168.203.158@tcp103 [8/256/0/180] [13099.492243] LNet: Added LNI 192.168.203.158@tcp104 [8/256/0/180] [13100.501825] LNet: Added LNI 192.168.203.158@tcp105 [8/256/0/180] [13101.467616] LNet: Added LNI 192.168.203.158@tcp106 [8/256/0/180] [13102.296339] LNet: Added LNI 192.168.203.158@tcp107 [8/256/0/180] [13103.012452] LNet: Added LNI 192.168.203.158@tcp108 [8/256/0/180] [13103.678665] LNet: Added LNI 192.168.203.158@tcp109 [8/256/0/180] [13104.588453] LNet: Added LNI 192.168.203.158@tcp110 [8/256/0/180] [13105.379895] LNet: Added LNI 192.168.203.158@tcp111 [8/256/0/180] [13106.151795] LNet: Added LNI 192.168.203.158@tcp112 [8/256/0/180] [13107.045637] LNet: Added LNI 192.168.203.158@tcp113 [8/256/0/180] [13107.960878] LNet: Added LNI 192.168.203.158@tcp114 [8/256/0/180] [13109.044536] LNet: Added LNI 192.168.203.158@tcp115 [8/256/0/180] [13109.997416] LNet: Added LNI 192.168.203.158@tcp116 [8/256/0/180] [13111.037086] LNet: Added LNI 192.168.203.158@tcp117 [8/256/0/180] [13111.907943] LNet: Added LNI 192.168.203.158@tcp118 [8/256/0/180] [13112.610709] LNet: Added LNI 192.168.203.158@tcp119 [8/256/0/180] [13113.569325] LNet: Added LNI 192.168.203.158@tcp120 [8/256/0/180] [13114.687712] LNet: Added LNI 192.168.203.158@tcp121 [8/256/0/180] [13115.715388] LNet: Added LNI 192.168.203.158@tcp122 [8/256/0/180] [13116.870624] LNet: Added LNI 192.168.203.158@tcp123 [8/256/0/180] [13118.249549] LNet: Added LNI 192.168.203.158@tcp124 [8/256/0/180] [13119.243262] LNet: Added LNI 192.168.203.158@tcp125 [8/256/0/180] [13119.917344] LNet: Added LNI 192.168.203.158@tcp126 [8/256/0/180] [13121.073532] LNet: Added LNI 192.168.203.158@tcp127 [8/256/0/180] [13121.852212] LNet: Added LNI 192.168.203.158@tcp128 [8/256/0/180] [13122.744450] LNet: Added LNI 192.168.203.158@tcp129 [8/256/0/180] [13123.610560] LNet: Added LNI 192.168.203.158@tcp130 [8/256/0/180] [13124.364401] LNet: Added LNI 192.168.203.158@tcp131 [8/256/0/180] [13125.466104] LNet: Added LNI 192.168.203.158@tcp132 [8/256/0/180] [13126.685880] LNet: Added LNI 192.168.203.158@tcp133 [8/256/0/180] [13127.829336] LNet: Added LNI 192.168.203.158@tcp134 [8/256/0/180] [13129.221151] LNet: Added LNI 192.168.203.158@tcp135 [8/256/0/180] [13136.330133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13136.363451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13137.489945] Lustre: 463433:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [13137.620397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13137.653345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13137.737748] Lustre: lustre-MDT0000: new disk, initializing [13137.849144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13137.868956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13141.876890] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13152.610380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13152.804714] Lustre: 463449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13152.870605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13152.888726] Lustre: Skipped 1 previous similar message [13152.984501] Lustre: lustre-MDT0001: new disk, initializing [13153.049307] Lustre: 464248:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [13153.058568] Lustre: 464248:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13153.261893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13153.302298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13153.313910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13158.884835] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13168.713485] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13174.302150] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13181.051389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13181.344083] Lustre: lustre-OST0000: new disk, initializing [13181.347073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13181.352895] Lustre: 465419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13181.443682] Lustre: 465424:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [13181.451669] Lustre: 465424:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13181.506643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13188.288785] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13189.683686] Lustre: 465844:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.158@tcp: can't add NID 192.168.203.158@tcp130: rc = -75 [13189.701459] Lustre: 465844:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13189.767382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13189.782968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13190.001577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13197.859785] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13210.598475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13210.604928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13210.634522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13215.715354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13215.723416] Lustre: Skipped 2 previous similar messages [13220.320163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13220.664578] Lustre: server umount lustre-OST0000 complete [13224.934521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13224.943118] LustreError: Skipped 1 previous similar message [13224.945892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13224.961956] Lustre: Skipped 1 previous similar message [13224.973024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13225.962330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13231.077961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13231.091866] Lustre: Skipped 3 previous similar messages [13236.194719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13236.208235] Lustre: Skipped 1 previous similar message [13239.264151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13239.452044] Lustre: server umount lustre-MDT0000 complete [13244.014673] LustreError: 463441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788553717 with bad export cookie 15160406098771668935 [13244.039181] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13244.458888] Lustre: server umount lustre-MDT0001 complete [13256.465454] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13259.542470] Key type lgssc unregistered [13260.034678] LNet: 467335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13260.051287] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13260.067985] LNet: Removed LNI 192.168.203.158@tcp [13261.088176] LNet: Removed LNI 192.168.203.158@tcp100 [13261.090424] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13261.099137] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13261.115825] LNet: Removed LNI 192.168.203.158@tcp101 [13262.176250] LNet: Removed LNI 192.168.203.158@tcp102 [13262.195181] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13262.206140] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13262.227129] LNet: Removed LNI 192.168.203.158@tcp103 [13263.264156] LNet: Removed LNI 192.168.203.158@tcp104 [13264.288145] LNet: Removed LNI 192.168.203.158@tcp105 [13264.295640] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13264.307984] LNetError: 467335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13264.315468] LNet: Removed LNI 192.168.203.158@tcp106 [13265.376627] LNet: Removed LNI 192.168.203.158@tcp107 [13266.400201] LNet: Removed LNI 192.168.203.158@tcp108 [13267.424149] LNet: Removed LNI 192.168.203.158@tcp109 [13267.426212] LNet: Removed LNI 192.168.203.158@tcp110 [13267.427873] LNet: Removed LNI 192.168.203.158@tcp111 [13267.429774] LNet: Removed LNI 192.168.203.158@tcp112 [13267.432973] LNet: Removed LNI 192.168.203.158@tcp113 [13267.449049] LNet: Removed LNI 192.168.203.158@tcp114 [13267.451419] LNet: Removed LNI 192.168.203.158@tcp115 [13267.456413] LNet: Removed LNI 192.168.203.158@tcp116 [13267.467980] LNet: Removed LNI 192.168.203.158@tcp117 [13267.475962] LNet: Removed LNI 192.168.203.158@tcp118 [13267.478975] LNet: Removed LNI 192.168.203.158@tcp119 [13267.483207] LNet: Removed LNI 192.168.203.158@tcp120 [13267.485535] LNet: Removed LNI 192.168.203.158@tcp121 [13267.488048] LNet: Removed LNI 192.168.203.158@tcp122 [13267.490120] LNet: Removed LNI 192.168.203.158@tcp123 [13267.492178] LNet: Removed LNI 192.168.203.158@tcp124 [13267.493806] LNet: Removed LNI 192.168.203.158@tcp125 [13267.496133] LNet: Removed LNI 192.168.203.158@tcp126 [13267.499293] LNet: Removed LNI 192.168.203.158@tcp127 [13267.501290] LNet: Removed LNI 192.168.203.158@tcp128 [13267.503415] LNet: Removed LNI 192.168.203.158@tcp129 [13267.505201] LNet: Removed LNI 192.168.203.158@tcp130 [13267.507708] LNet: Removed LNI 192.168.203.158@tcp131 [13267.510270] LNet: Removed LNI 192.168.203.158@tcp132 [13267.512061] LNet: Removed LNI 192.168.203.158@tcp133 [13267.525935] LNet: Removed LNI 192.168.203.158@tcp134 [13267.538086] LNet: Removed LNI 192.168.203.158@tcp135 [13268.274890] Key type .llcrypt unregistered [13268.277770] Key type ._llcrypt unregistered [13302.033822] Key type ._llcrypt registered [13302.038378] Key type .llcrypt registered [13302.239762] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [13322.059332] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13323.402598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13323.442833] alg: No test for adler32 (adler32-zlib) [13324.639443] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [13324.990323] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [13326.784210] Key type lgssc registered [13328.053026] Lustre: Echo OBD driver; http://www.lustre.org/ [13361.824682] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:30:34 (1788553834) [13380.903609] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13384.244919] Key type lgssc unregistered [13384.590728] LNet: 471423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13384.595544] LNetError: 471423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13384.616884] LNet: Removed LNI 192.168.203.158@tcp [13385.732907] Key type .llcrypt unregistered [13385.741541] Key type ._llcrypt unregistered [13404.955378] Key type ._llcrypt registered [13404.958808] Key type .llcrypt registered [13405.101505] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13406.334063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13406.410753] alg: No test for adler32 (adler32-zlib) [13407.606512] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [13407.850272] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [13409.520530] Key type lgssc registered [13410.790406] Lustre: Echo OBD driver; http://www.lustre.org/ [13420.707592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13420.737894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13422.125285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13422.179438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13422.274244] Lustre: lustre-MDT0000: new disk, initializing [13422.370434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13422.388447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13427.158211] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13437.871110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13438.049291] Lustre: 472634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13438.123194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13438.130483] Lustre: Skipped 1 previous similar message [13438.171136] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13438.220616] Lustre: lustre-MDT0001: new disk, initializing [13438.314565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13438.359631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13438.367046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13443.848802] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13453.419661] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13458.762467] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13465.588334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13465.915892] Lustre: lustre-OST0000: new disk, initializing [13465.923282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13465.928860] Lustre: 474607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13466.042128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13472.841238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13472.851421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13473.081186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13473.468964] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13483.431518] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13489.200129] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing lnet_if_list [13495.241485] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13495.266750] LNet: Added LNI 192.168.203.158@tcp10 [8/256/0/180] [13501.034342] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13501.061575] LNet: Added LNI 192.168.203.158@tcp11 [8/256/0/180] [13505.326773] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13505.359966] LNet: Added LNI 192.168.203.158@tcp12 [8/256/0/180] [13510.510423] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13510.564644] LNet: Added LNI 192.168.203.158@tcp13 [8/256/0/180] [13515.849837] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13515.877885] LNet: Added LNI 192.168.203.158@tcp14 [8/256/0/180] [13521.593978] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13521.632695] LNet: Added LNI 192.168.203.158@tcp15 [8/256/0/180] [13527.432445] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13527.495334] LNet: Added LNI 192.168.203.158@tcp16 [8/256/0/180] [13533.271662] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13533.323756] LNet: Added LNI 192.168.203.158@tcp17 [8/256/0/180] [13539.139462] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13539.169832] LNet: Added LNI 192.168.203.158@tcp18 [8/256/0/180] [13544.380798] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13544.406456] LNet: Added LNI 192.168.203.158@tcp19 [8/256/0/180] [13549.515225] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13549.569333] LNet: Added LNI 192.168.203.158@tcp20 [8/256/0/180] [13555.487379] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13555.515255] LNet: Added LNI 192.168.203.158@tcp21 [8/256/0/180] [13562.036294] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13562.103832] LNet: Added LNI 192.168.203.158@tcp22 [8/256/0/180] [13568.518239] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13568.540742] LNet: Added LNI 192.168.203.158@tcp23 [8/256/0/180] [13574.649247] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13574.676628] LNet: Added LNI 192.168.203.158@tcp24 [8/256/0/180] [13580.559373] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13580.601572] LNet: Added LNI 192.168.203.158@tcp25 [8/256/0/180] [13587.230844] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13587.285287] LNet: Added LNI 192.168.203.158@tcp26 [8/256/0/180] [13594.122402] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13594.149944] LNet: Added LNI 192.168.203.158@tcp27 [8/256/0/180] [13600.729915] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13600.776859] LNet: Added LNI 192.168.203.158@tcp28 [8/256/0/180] [13607.446388] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13607.476717] LNet: Added LNI 192.168.203.158@tcp29 [8/256/0/180] [13614.138774] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13614.214792] LNet: Added LNI 192.168.203.158@tcp30 [8/256/0/180] [13621.270641] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13621.294903] LNet: Added LNI 192.168.203.158@tcp31 [8/256/0/180] [13628.555320] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13628.590708] LNet: Added LNI 192.168.203.158@tcp32 [8/256/0/180] [13636.043781] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13636.069692] LNet: Added LNI 192.168.203.158@tcp33 [8/256/0/180] [13642.996200] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13643.029924] LNet: Added LNI 192.168.203.158@tcp34 [8/256/0/180] [13650.399488] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13650.451711] LNet: Added LNI 192.168.203.158@tcp35 [8/256/0/180] [13657.361654] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13657.386426] LNet: Added LNI 192.168.203.158@tcp36 [8/256/0/180] [13664.284685] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13664.322463] LNet: Added LNI 192.168.203.158@tcp37 [8/256/0/180] [13671.736455] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13671.780935] LNet: Added LNI 192.168.203.158@tcp38 [8/256/0/180] [13679.989596] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13680.041818] LNet: Added LNI 192.168.203.158@tcp39 [8/256/0/180] [13688.141193] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13688.169076] LNet: Added LNI 192.168.203.158@tcp40 [8/256/0/180] [13696.210025] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13696.254296] LNet: Added LNI 192.168.203.158@tcp41 [8/256/0/180] [13705.070261] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13705.113446] LNet: Added LNI 192.168.203.158@tcp42 [8/256/0/180] [13713.811978] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13713.852987] LNet: Added LNI 192.168.203.158@tcp43 [8/256/0/180] [13722.932771] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13722.971681] LNet: Added LNI 192.168.203.158@tcp44 [8/256/0/180] [13731.184891] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13731.211550] LNet: Added LNI 192.168.203.158@tcp45 [8/256/0/180] [13740.158443] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13740.187033] LNet: Added LNI 192.168.203.158@tcp46 [8/256/0/180] [13749.086821] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13749.137749] LNet: Added LNI 192.168.203.158@tcp47 [8/256/0/180] [13758.001239] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13758.058778] LNet: Added LNI 192.168.203.158@tcp48 [8/256/0/180] [13767.405471] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13767.444607] LNet: Added LNI 192.168.203.158@tcp49 [8/256/0/180] [13777.206665] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13777.241687] LNet: Added LNI 192.168.203.158@tcp50 [8/256/0/180] [13796.839428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13796.849969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13796.860307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13800.933356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13800.934673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13802.632809] Lustre: server umount lustre-MDT0000 complete [13806.496049] LustreError: 472629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788554280 with bad export cookie 1945270086098520596 [13806.511131] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13806.518061] LustreError: 472629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13806.946951] Lustre: server umount lustre-MDT0001 complete [13826.509373] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13827.552670] Lustre: 471908:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788554285/real 1788554285] req@000000004151b2b5 x1875434674505344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788554301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13827.578941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13827.594695] Lustre: Skipped 1 previous similar message [13832.737106] Lustre: 471908:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788554290/real 1788554290] req@00000000c41c5c00 x1875434674505472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788554306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13835.078929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13837.795937] Lustre: 471905:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788554295/real 1788554295] req@000000006770bb71 x1875434674505728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788554311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13842.976313] Lustre: 471905:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788554300/real 1788554300] req@000000009b844c71 x1875434674505856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788554316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13851.811991] LustreError: 474600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13851.860364] LustreError: 474600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13851.915753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13852.960895] LustreError: 474601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13852.994976] LustreError: 474601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13856.050640] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13856.082878] LustreError: 498848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.154209] LustreError: 498848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13858.521627] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13861.353920] LustreError: 498846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.406551] LustreError: 498846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13866.469031] LustreError: 498848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13866.495044] LustreError: 498848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13870.407637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13870.675279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13875.705323] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13876.083087] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [13883.870254] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13890.388920] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13911.526325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13911.535320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13911.547754] Lustre: Skipped 1 previous similar message [13911.558332] Lustre: Skipped 2 previous similar messages [13911.577221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13915.668374] Lustre: server umount lustre-OST0000 complete [13921.763416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13921.766421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13921.766841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13921.779845] Lustre: Skipped 1 previous similar message [13921.786700] Lustre: Skipped 1 previous similar message [13925.583680] Lustre: server umount lustre-MDT0000 complete [13926.885526] LustreError: 498851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13926.914647] LustreError: 498851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13928.176699] LustreError: 472628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788554402 with bad export cookie 1945270086098523284 [13928.179182] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13928.183473] LustreError: 472628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13928.512923] Lustre: server umount lustre-MDT0001 complete [13937.866297] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [13940.580672] Key type lgssc unregistered [13940.925894] LNet: 501650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13940.933474] LNetError: 501650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13940.942660] LNet: Removed LNI 192.168.203.158@tcp [13940.949756] LNet: Removed LNI 192.168.203.158@tcp10 [13940.960486] LNet: Removed LNI 192.168.203.158@tcp11 [13940.966879] LNet: Removed LNI 192.168.203.158@tcp12 [13940.976570] LNet: Removed LNI 192.168.203.158@tcp13 [13940.983418] LNet: Removed LNI 192.168.203.158@tcp14 [13940.988211] LNet: Removed LNI 192.168.203.158@tcp15 [13940.989892] LNet: Removed LNI 192.168.203.158@tcp16 [13940.991808] LNet: Removed LNI 192.168.203.158@tcp17 [13940.998050] LNet: Removed LNI 192.168.203.158@tcp18 [13941.004448] LNet: Removed LNI 192.168.203.158@tcp19 [13941.007142] LNet: Removed LNI 192.168.203.158@tcp20 [13941.011859] LNet: Removed LNI 192.168.203.158@tcp21 [13941.018561] LNet: Removed LNI 192.168.203.158@tcp22 [13941.021362] LNet: Removed LNI 192.168.203.158@tcp23 [13941.024247] LNet: Removed LNI 192.168.203.158@tcp24 [13941.027507] LNet: Removed LNI 192.168.203.158@tcp25 [13941.029914] LNet: Removed LNI 192.168.203.158@tcp26 [13941.033258] LNet: Removed LNI 192.168.203.158@tcp27 [13941.035848] LNet: Removed LNI 192.168.203.158@tcp28 [13941.043657] LNet: Removed LNI 192.168.203.158@tcp29 [13941.045623] LNet: Removed LNI 192.168.203.158@tcp30 [13941.051890] LNet: Removed LNI 192.168.203.158@tcp31 [13941.056391] LNet: Removed LNI 192.168.203.158@tcp32 [13941.062881] LNet: Removed LNI 192.168.203.158@tcp33 [13941.064868] LNet: Removed LNI 192.168.203.158@tcp34 [13941.070315] LNet: Removed LNI 192.168.203.158@tcp35 [13941.073299] LNet: Removed LNI 192.168.203.158@tcp36 [13941.083381] LNet: Removed LNI 192.168.203.158@tcp37 [13941.093878] LNet: Removed LNI 192.168.203.158@tcp38 [13941.105053] LNet: Removed LNI 192.168.203.158@tcp39 [13941.121111] LNet: Removed LNI 192.168.203.158@tcp40 [13941.123291] LNet: Removed LNI 192.168.203.158@tcp41 [13941.127824] LNet: Removed LNI 192.168.203.158@tcp42 [13941.136220] LNet: Removed LNI 192.168.203.158@tcp43 [13941.139103] LNet: Removed LNI 192.168.203.158@tcp44 [13941.144732] LNet: Removed LNI 192.168.203.158@tcp45 [13941.148738] LNet: Removed LNI 192.168.203.158@tcp46 [13941.162077] LNet: Removed LNI 192.168.203.158@tcp47 [13941.164585] LNet: Removed LNI 192.168.203.158@tcp48 [13941.170313] LNet: Removed LNI 192.168.203.158@tcp49 [13941.188072] LNet: Removed LNI 192.168.203.158@tcp50 [13941.787746] Key type .llcrypt unregistered [13941.789464] Key type ._llcrypt unregistered [13962.853569] Key type ._llcrypt registered [13962.855457] Key type .llcrypt registered [13962.992775] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [13978.253176] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [13979.283989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13979.328186] alg: No test for adler32 (adler32-zlib) [13980.371347] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [13980.638786] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [13982.320447] Key type lgssc registered [13983.387855] Lustre: Echo OBD driver; http://www.lustre.org/ [14019.376824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14019.414816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14020.750959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14020.805620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14020.916393] Lustre: lustre-MDT0000: new disk, initializing [14021.033135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14021.054179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14024.648581] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14033.959444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14034.073785] Lustre: 504852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14034.120710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14034.124843] Lustre: Skipped 1 previous similar message [14034.206130] Lustre: lustre-MDT0001: new disk, initializing [14034.348540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14034.386042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14034.390486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14038.612995] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14048.777869] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14054.216179] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14062.101747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14062.300593] Lustre: *** cfs_fail_loc=90f, val=0*** [14062.403382] Lustre: lustre-OST0000: new disk, initializing [14062.420357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14062.440963] Lustre: 506867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14062.556085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14069.322877] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14069.324666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14069.344561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14069.498521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14079.157933] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14090.230866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14090.234360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14090.244070] LustreError: Skipped 1 previous similar message [14090.245869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14090.270077] Lustre: Skipped 1 previous similar message [14091.777035] Lustre: server umount lustre-OST0000 complete [14095.851895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14095.863493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14095.888064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14095.901899] Lustre: Skipped 1 previous similar message [14100.450033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14100.452157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14100.473039] Lustre: Skipped 1 previous similar message [14102.003727] Lustre: server umount lustre-MDT0000 complete [14105.937122] LustreError: 505645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788554579 with bad export cookie 15663065119285781094 [14105.951746] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14105.952199] LustreError: 505645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14106.357561] Lustre: server umount lustre-MDT0001 complete [14117.182791] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [14119.796448] Key type lgssc unregistered [14120.043883] LNet: 508770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14120.051801] LNetError: 508770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14120.072788] LNet: Removed LNI 192.168.203.158@tcp [14120.869749] Key type .llcrypt unregistered [14120.871514] Key type ._llcrypt unregistered [14139.451978] Key type ._llcrypt registered [14139.454359] Key type .llcrypt registered [14139.550705] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [14140.409884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14140.508380] alg: No test for adler32 (adler32-zlib) [14141.528422] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [14141.780043] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [14143.464369] Key type lgssc registered [14144.543798] Lustre: Echo OBD driver; http://www.lustre.org/ [14151.496669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14151.517266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14153.023061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14157.289375] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14162.406682] LustreError: 510034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14162.435031] LustreError: 510034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14165.500063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14165.870777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14170.448241] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14177.758448] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14183.182307] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14189.321909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14189.592476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14195.669548] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14202.181981] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14215.826609] loop0: detected capacity change from 0 to 204800000 [14216.191277] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14216.212913] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14216.236496] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14223.920629] loop0: detected capacity change from 0 to 204800000 [14224.131601] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14224.141955] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14224.154936] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14229.762140] loop0: detected capacity change from 0 to 204800000 [14234.936597] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14235.119634] Lustre: fs15246-OST0000: new disk, initializing [14235.123572] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14235.130471] Lustre: 513625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14235.185960] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14241.052853] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14249.428930] loop1: detected capacity change from 0 to 204800000 [14254.641663] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14254.761942] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14254.834151] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14254.920560] Lustre: fs15246-MDT0000: new disk, initializing [14255.055973] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14255.103530] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14255.238731] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14256.290970] Lustre: 514749:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c3b263b0 x1875435444026624/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788554746 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14259.276730] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14265.391189] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14265.408300] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14265.426075] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14277.097057] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14277.105961] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14282.214327] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14282.897220] Lustre: server umount fs15246-MDT0000 complete [14287.342851] LustreError: 510029:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14290.738810] Lustre: server umount fs15246-OST0000 complete [14302.692423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14302.707323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14302.713777] Lustre: Skipped 1 previous similar message [14302.726093] Lustre: Skipped 1 previous similar message [14306.973210] Lustre: server umount lustre-OST0000 complete [14312.929835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14312.944197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14312.946291] Lustre: Skipped 1 previous similar message [14312.960059] Lustre: Skipped 1 previous similar message [14316.724421] Lustre: server umount lustre-MDT0000 complete [14318.051165] LustreError: 510034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14318.069369] LustreError: 510034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14320.012183] LustreError: 514743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788554793 with bad export cookie 1681445697097313898 [14320.013858] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14320.035159] LustreError: 514743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14320.344871] Lustre: server umount lustre-MDT0001 complete [14328.570945] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [14330.887852] Key type lgssc unregistered [14331.167796] LNet: 517844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14331.181416] LNetError: 517844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14331.197597] LNet: Removed LNI 192.168.203.158@tcp [14331.989644] Key type .llcrypt unregistered [14331.991738] Key type ._llcrypt unregistered [14348.310880] Key type ._llcrypt registered [14348.312928] Key type .llcrypt registered [14348.396340] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [14349.256812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14349.303208] alg: No test for adler32 (adler32-zlib) [14350.354981] Lustre: Lustre: Build Version: 2.17.58_2_g4093f3c [14350.595221] LNet: Added LNI 192.168.203.158@tcp [8/256/0/180] [14352.248184] Key type lgssc registered [14353.086791] Lustre: Echo OBD driver; http://www.lustre.org/ [14388.311360] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [14399.329155] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [14429.536066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14429.581206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14430.872797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14430.908486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14430.986921] Lustre: lustre-MDT0000: new disk, initializing [14431.058761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14431.087107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14434.924687] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14443.794969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14443.884715] Lustre: 522298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14443.944529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14443.947768] Lustre: Skipped 1 previous similar message [14444.012326] Lustre: lustre-MDT0001: new disk, initializing [14444.099548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14444.164404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14444.173797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14448.264528] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14457.931522] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14464.027841] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14471.699554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14471.978700] Lustre: lustre-OST0000: new disk, initializing [14471.983219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14471.994960] Lustre: 524267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14472.138340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14477.558809] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_default_debug -1 all [14482.054936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14482.064182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14482.178925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14485.829723] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14489.776214] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14489.972816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14494.291124] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14494.464171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14497.768908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14497.770566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14497.773619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14497.785622] LustreError: Skipped 1 previous similar message [14497.824933] Lustre: Skipped 1 previous similar message [14502.535207] Lustre: server umount lustre-OST0000 complete [14508.006903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14508.007891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14508.025502] Lustre: Skipped 1 previous similar message [14508.050855] Lustre: Skipped 2 previous similar messages [14511.010236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14511.028607] Lustre: Skipped 1 previous similar message [14516.193083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14516.204703] Lustre: Skipped 1 previous similar message [14520.288111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14520.395249] Lustre: server umount lustre-MDT0000 complete [14523.716320] LustreError: 523094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788554997 with bad export cookie 2869697025157920849 [14523.725203] LustreError: MGC192.168.203.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14523.728780] LustreError: 523094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14524.080107] Lustre: server umount lustre-MDT0001 complete [14548.007454] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing set_hostid [14558.396119] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing load_modules_local [14586.036105] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14336 sec ========= 16:50:58 (1788555058) [14587.506690] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:51:00 (1788555060) === [14590.407982] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:51:03 (1788555063) === [14625.795324] Lustre: DEBUG MARKER: oleg358-server.virtnet: executing unload_modules_local [14628.182695] Key type lgssc unregistered [14628.442596] LNet: 530908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14628.452746] LNetError: 530908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14628.464869] LNet: Removed LNI 192.168.203.158@tcp [14629.249894] Key type .llcrypt unregistered [14629.253969] Key type ._llcrypt unregistered