[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471234675 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003176] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008373] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022440] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026750] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030087] *** VALIDATE tmpfs *** [ 0.031345] *** VALIDATE proc *** [ 0.032166] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.035211] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042458] debug: unmapping init [mem 0xffffffff88859000-0xffffffff88860fff] [ 0.044185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050021] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053313] rcu: Hierarchical SRCU implementation. [ 0.055586] smp: Bringing up secondary CPUs ... [ 0.056662] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060338] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142307] node 0 deferred pages initialised in 77ms [ 0.146123] devtmpfs: initialized [ 0.147205] x86/mm: Memory block size: 128MB [ 0.149670] gcov: version magic: 0x41383552 [ 0.150586] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155256] pinctrl core: initialized pinctrl subsystem [ 0.157161] [ 0.157435] ************************************************************* [ 0.160010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162010] ** ** [ 0.164012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165021] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.169011] ** IOMMU data structures, which may compromise security on ** [ 0.171014] ** your system. ** [ 0.174010] ** ** [ 0.175007] ** If you see this message and you are not debugging the ** [ 0.177012] ** kernel, report this immediately to your vendor! ** [ 0.179012] ** ** [ 0.181010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183011] ************************************************************* [ 0.185631] NET: Registered protocol family 16 [ 0.187419] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195080] cpuidle: using governor menu [ 0.196694] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199484] PCI: Using configuration type 1 for base access [ 0.201126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216085] cryptd: max_cpu_qlen set to 1000 [ 0.218308] ACPI: Added _OSI(Module Device) [ 0.220017] ACPI: Added _OSI(Processor Device) [ 0.222016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228924] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235363] ACPI: Interpreter enabled [ 0.237067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239014] ACPI: Using IOAPIC for interrupt routing [ 0.241102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255009] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267654] acpiphp: Slot [2] registered [ 0.269112] acpiphp: Slot [5] registered [ 0.270138] acpiphp: Slot [6] registered [ 0.272110] acpiphp: Slot [7] registered [ 0.273141] acpiphp: Slot [8] registered [ 0.275109] acpiphp: Slot [9] registered [ 0.276141] acpiphp: Slot [10] registered [ 0.277142] acpiphp: Slot [3] registered [ 0.279110] acpiphp: Slot [4] registered [ 0.280146] acpiphp: Slot [11] registered [ 0.282153] acpiphp: Slot [12] registered [ 0.283132] acpiphp: Slot [13] registered [ 0.285156] acpiphp: Slot [14] registered [ 0.286119] acpiphp: Slot [15] registered [ 0.288144] acpiphp: Slot [16] registered [ 0.289127] acpiphp: Slot [17] registered [ 0.291155] acpiphp: Slot [18] registered [ 0.292141] acpiphp: Slot [19] registered [ 0.294136] acpiphp: Slot [20] registered [ 0.295126] acpiphp: Slot [21] registered [ 0.297131] acpiphp: Slot [22] registered [ 0.298118] acpiphp: Slot [23] registered [ 0.299175] acpiphp: Slot [24] registered [ 0.301140] acpiphp: Slot [25] registered [ 0.302118] acpiphp: Slot [26] registered [ 0.304151] acpiphp: Slot [27] registered [ 0.305140] acpiphp: Slot [28] registered [ 0.307155] acpiphp: Slot [29] registered [ 0.311216] acpiphp: Slot [30] registered [ 0.312118] acpiphp: Slot [31] registered [ 0.314122] PCI host bridge to bus 0000:00 [ 0.315032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323048] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334044] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337181] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348671] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368393] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371967] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378995] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408886] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455551] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494715] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.561015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.570285] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.603018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.610310] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.630013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.646016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.656325] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.660387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.662466] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.664368] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.667287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.672160] iommu: Default domain type: Passthrough [ 0.674379] SCSI subsystem initialized [ 0.676141] ACPI: bus type USB registered [ 0.677122] usbcore: registered new interface driver usbfs [ 0.679114] usbcore: registered new interface driver hub [ 0.682084] usbcore: registered new device driver usb [ 0.684169] pps_core: LinuxPPS API ver. 1 registered [ 0.686012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689057] PTP clock support registered [ 0.691133] EDAC MC: Ver: 3.0.0 [ 0.693124] PCI: Using ACPI for IRQ routing [ 0.696504] NetLabel: Initializing [ 0.697026] NetLabel: domain hash size = 128 [ 0.699012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.701081] NetLabel: unlabeled traffic allowed by default [ 0.703189] vgaarb: loaded [ 0.705281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.707017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715354] clocksource: Switched to clocksource kvm-clock [ 0.816489] VFS: Disk quotas dquot_6.6.0 [ 0.818765] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821250] *** VALIDATE ramfs *** [ 0.822470] *** VALIDATE hugetlbfs *** [ 0.823967] pnp: PnP ACPI init [ 0.826251] pnp: PnP ACPI: found 6 devices [ 0.874882] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.878523] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.880886] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.883103] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885699] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.888397] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891431] NET: Registered protocol family 2 [ 0.894163] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.900330] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904235] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.909807] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913939] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917069] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.920686] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923906] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927232] NET: Registered protocol family 1 [ 0.930429] RPC: Registered named UNIX socket transport module. [ 0.932881] RPC: Registered udp transport module. [ 0.934771] RPC: Registered tcp transport module. [ 0.936824] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939276] NET: Registered protocol family 44 [ 0.940932] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943133] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.945350] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.947708] PCI: CLS 0 bytes, default 64 [ 0.949698] Unpacking initramfs... [ 2.352499] debug: unmapping init [mem 0xffff9456bcc54000-0xffff9456bffbffff] [ 2.357361] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.359862] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.363065] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.897682] Initialise system trusted keyrings [ 2.899675] Key type blacklist registered [ 2.902032] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.912340] zbud: loaded [ 2.915573] *** VALIDATE nfs *** [ 2.917289] *** VALIDATE nfs4 *** [ 2.918900] pstore: using deflate compression [ 2.922143] Platform Keyring initialized [ 3.024489] NET: Registered protocol family 38 [ 3.026403] Key type asymmetric registered [ 3.028146] Asymmetric key parser 'x509' registered [ 3.030122] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.033595] io scheduler mq-deadline registered [ 3.035569] io scheduler kyber registered [ 3.037366] io scheduler bfq registered [ 3.039032] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.041851] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.045059] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.048307] ACPI: Power Button [PWRF] [ 3.055415] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.062725] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.077478] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.085641] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.115290] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.144877] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.171926] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.176412] Non-volatile memory driver v1.3 [ 3.178323] Linux agpgart interface v0.103 [ 3.216111] virtio_blk virtio1: [vda] 146744 512-byte logical blocks (75.1 MB/71.7 MiB) [ 3.220610] vda: detected capacity change from 0 to 75132928 [ 3.239039] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.242189] vdb: detected capacity change from 0 to 1073741824 [ 3.259614] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.262831] vdc: detected capacity change from 0 to 2621440000 [ 3.280469] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.284328] vdd: detected capacity change from 0 to 2621440000 [ 3.303552] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.306565] vde: detected capacity change from 0 to 4294967296 [ 3.329173] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.332151] vdf: detected capacity change from 0 to 4294967296 [ 3.340684] libphy: Fixed MDIO Bus: probed [ 3.347372] usbcore: registered new interface driver usbserial_generic [ 3.349471] usbserial: USB Serial support registered for generic [ 3.351520] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.356273] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.358040] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.360143] mousedev: PS/2 mouse device common for all mice [ 3.363282] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.364766] rtc_cmos 00:05: RTC can wake from S4 [ 3.370094] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.370436] rtc_cmos 00:05: registered as rtc0 [ 3.374697] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.376129] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.377327] intel_pstate: CPU model not supported [ 3.382573] hid: raw HID events driver (C) Jiri Kosina [ 3.385122] usbcore: registered new interface driver usbhid [ 3.387356] usbhid: USB HID core driver [ 3.388989] drop_monitor: Initializing network drop monitor service [ 3.392203] Initializing XFRM netlink socket [ 3.394491] NET: Registered protocol family 10 [ 3.398240] Segment Routing with IPv6 [ 3.400026] NET: Registered protocol family 17 [ 3.402660] mpls_gso: MPLS GSO support [ 3.407855] RAS: Correctable Errors collector initialized. [ 3.409881] AVX version of gcm_enc/dec engaged. [ 3.411637] AES CTR mode by8 optimization enabled [ 3.488619] sched_clock: Marking stable (3488522329, 0)->(4376748238, -888225909) [ 3.492351] registered taskstats version 1 [ 3.494579] Loading compiled-in X.509 certificates [ 3.496716] zswap: loaded using pool lzo/zbud [ 3.523535] Key type big_key registered [ 3.536536] Key type encrypted registered [ 3.538457] ima: No TPM chip found, activating TPM-bypass! [ 3.540658] ima: Allocated hash algorithm: sha1 [ 3.542413] ima: No architecture policies found [ 3.543975] evm: Initialising EVM extended attributes: [ 3.545954] evm: security.selinux [ 3.547327] evm: security.ima [ 3.548463] evm: security.capability [ 3.549895] evm: HMAC attrs: 0x1 [ 3.552528] rtc_cmos 00:05: setting system clock to 2026-09-08 04:00:57 UTC (1788840057) [ 3.560160] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.563433] debug: unmapping init [mem 0xffffffff88582000-0xffffffff88858fff] [ 3.574178] Write protecting the kernel read-only data: 28672k [ 3.577895] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 3.580719] debug: unmapping init [mem 0xffffffff87514000-0xffffffff875fffff] [ 3.618741] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.627493] systemd[1]: Detected virtualization kvm. [ 3.629492] systemd[1]: Detected architecture x86-64. [ 3.631365] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.663324] systemd[1]: No hostname configured. [ 3.665168] systemd[1]: Set hostname to . [ 3.667471] random: systemd: uninitialized urandom read (16 bytes read) [ 3.670286] systemd[1]: Initializing machine ID from random generator. [ 3.720212] random: ln: uninitialized urandom read (6 bytes read) [ 3.823470] random: systemd: uninitialized urandom read (16 bytes read) [ 3.826973] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.832401] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.837821] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.494500] device-mapper: uevent: version 1.0.3 [ 4.496897] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK [ 4.734701] random: fast init done ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.232827] virtio_net virtio0 ens2: renamed from eth0 [ 5.273656] scsi host0: ata_piix [ 5.311508] scsi host1: ata_piix [ 5.313803] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.316109] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.715212] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.056660] random: crng init done [ 10.058102] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.495108] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.647752] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.927086] SELinux: Disabled at runtime. [ 11.990350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.998691] systemd[1]: Detected virtualization kvm. [ 12.000234] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.488730] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.492487] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.498530] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.503707] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.506727] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.517436] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.529876] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.571818] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.015452] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.396334] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.459607] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.605322] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.622899] EDAC sbridge: Ver: 1.1.2 [ 15.156611] Key type dns_resolver registered [ 15.464419] NFS: Registering the id_resolver key type [ 15.466873] Key type id_resolver registered [ 15.468438] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg622-server login: [ 73.998378] libcfs: loading out-of-tree module taints kernel. [ 74.073613] Key type ._llcrypt registered [ 74.076806] Key type .llcrypt registered [ 74.199538] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 74.453113] hrtimer: interrupt took 2989761 ns [ 92.473991] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 94.363082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.389274] alg: No test for adler32 (adler32-zlib) [ 95.774705] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 96.632476] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 98.440135] Key type lgssc registered [ 100.769650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.843081] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.732631] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 172.523470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.549304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.735498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.752599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.808570] Lustre: lustre-MDT0000: new disk, initializing [ 173.873439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.890198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.779300] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.404233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.522619] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.577756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.586996] Lustre: Skipped 1 previous similar message [ 190.676148] Lustre: lustre-MDT0001: new disk, initializing [ 190.747056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.792746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.808490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.191298] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.832488] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.894835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.102809] Lustre: lustre-OST0000: new disk, initializing [ 209.109735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.119874] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.185908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.403835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.415332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.504930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 215.161858] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.662143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.941627] Lustre: lustre-OST0001: new disk, initializing [ 230.951728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 230.979329] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.085184] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.756986] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.662695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.668726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.725621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.241278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.032962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.066909] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.134639] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing yml_node [ 272.016306] Lustre: DEBUG MARKER: Client: 2.17.57.115 [ 274.822890] Lustre: DEBUG MARKER: MDS: 2.17.57.115 [ 277.515679] Lustre: DEBUG MARKER: OSS: 2.17.57.115 [ 279.402576] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 00:05:31 EDT 2026 [ 295.529931] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.987482] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.825266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.855194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.872069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.360329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.363624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.394395] Lustre: Skipped 3 previous similar messages [ 320.480264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.484778] Lustre: Skipped 2 previous similar messages [ 325.604993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.613115] Lustre: Skipped 3 previous similar messages [ 327.135088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 327.380150] Lustre: server umount lustre-MDT0000 complete [ 331.281518] LustreError: 9519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840385 with bad export cookie 4026593593838794813 [ 331.286436] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.299334] LustreError: 9519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 335.840332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.840642] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.842120] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.862948] Lustre: Skipped 3 previous similar messages [ 335.879205] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 336.865158] LustreError: 10152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.887661] LustreError: 10152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 337.967381] Lustre: server umount lustre-MDT0001 complete [ 341.885879] Lustre: server umount lustre-OST0000 complete [ 346.785414] Lustre: server umount lustre-OST0001 complete [ 353.349775] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 362.642888] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 396.609649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.850675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.884872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.987754] Lustre: lustre-MDT0000: new disk, initializing [ 397.066696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.094941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.196342] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 410.733743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.887427] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.967898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.970757] Lustre: Skipped 1 previous similar message [ 411.071716] Lustre: lustre-MDT0001: new disk, initializing [ 411.171065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.228717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 411.249632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 415.074338] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 424.476257] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 430.285966] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.421699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.726171] Lustre: lustre-OST0000: new disk, initializing [ 436.728617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 436.733887] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 436.803864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.834405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 438.859540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 438.969075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 442.267803] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 451.626953] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 457.319822] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 457.563437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.566205] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.768108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 464.875734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.882310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.883665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.893036] Lustre: Skipped 1 previous similar message [ 464.900461] Lustre: Skipped 3 previous similar messages [ 471.082260] Lustre: server umount lustre-OST0000 complete [ 475.105692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.133269] Lustre: Skipped 1 previous similar message [ 481.297082] Lustre: server umount lustre-MDT0000 complete [ 484.968801] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840538 with bad export cookie 4026593593838796346 [ 484.981458] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.982077] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 485.688218] Lustre: server umount lustre-MDT0001 complete [ 490.073460] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:09:02 (1788840542) [ 492.372923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 494.118765] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:09:06 (1788840546) [ 500.741328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.533633] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.613696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.870836] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.928526] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 511.973603] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.322887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.524897] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 529.438797] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.071722] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.395716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.050486] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 556.966986] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.361021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.374653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.378052] LustreError: Skipped 1 previous similar message [ 571.380427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.400291] Lustre: Skipped 1 previous similar message [ 571.427711] Lustre: Skipped 8 previous similar messages [ 576.220688] Lustre: server umount lustre-OST0000 complete [ 587.442301] Lustre: server umount lustre-MDT0000 complete [ 591.817129] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840645 with bad export cookie 4026593593838797228 [ 591.822348] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.848747] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 592.135342] Lustre: server umount lustre-MDT0001 complete [ 605.383697] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 608.684773] Key type lgssc unregistered [ 609.107765] LNet: 23005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.121857] LNetError: 23005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.143364] LNet: Removed LNI 192.168.206.122@tcp [ 610.040693] Key type .llcrypt unregistered [ 610.042694] Key type ._llcrypt unregistered [ 632.029297] Key type ._llcrypt registered [ 632.031560] Key type .llcrypt registered [ 632.137989] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 633.143026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.259310] alg: No test for adler32 (adler32-zlib) [ 634.336202] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 634.609150] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 636.311184] Key type lgssc registered [ 637.246724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.790214] loop: module loaded [ 644.862762] loop0: detected capacity change from 0 to 204800000 [ 645.051922] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.064135] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.313326] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.173360] loop0: detected capacity change from 0 to 204800000 [ 651.314173] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.332943] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.348261] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.273234] loop0: detected capacity change from 0 to 204800000 [ 661.413837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.457362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.737346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 662.812126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 662.891628] Lustre: lustre-MDT0000: new disk, initializing [ 662.978884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.993471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 666.635389] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 673.891607] loop1: detected capacity change from 0 to 204800000 [ 678.294713] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.485481] Lustre: lustre-OST07c6: new disk, initializing [ 678.489243] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 678.494119] Lustre: Skipped 1 previous similar message [ 678.499128] Lustre: 26185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 678.565374] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 682.012315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 682.024278] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 682.057462] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 683.383982] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 688.290861] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 688.302159] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 688.307439] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 688.313447] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 688.321494] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 688.328198] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 703.345099] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 703.355144] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 703.361840] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.373121] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 703.380155] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 703.390323] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.905977] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 707.913122] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.918554] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 707.924842] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.929820] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.935826] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.942350] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 707.948138] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.952373] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.956339] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.959857] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.963725] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.137830] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 711.149034] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 711.157237] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 711.163087] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.167912] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.177077] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.185224] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 711.192239] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.201081] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.208722] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.217726] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.221712] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.740360] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 715.755194] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 715.762202] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 715.772560] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.780188] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 715.790743] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.795653] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 715.802132] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.807309] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 715.814755] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 715.820356] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 715.825661] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.365329] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 725.381697] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 725.390399] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 725.405028] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.413340] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.418111] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.425273] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 725.433163] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.439909] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.448647] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.453752] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.460451] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.116131] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.131350] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.145242] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.233700] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 780.574260] Lustre: server umount lustre-OST07c6 complete [ 789.602803] Lustre: server umount lustre-MDT0000 complete [ 804.400714] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 806.730321] Key type lgssc unregistered [ 806.978049] LNet: 28741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.985254] LNetError: 28741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.042950] LNet: Removed LNI 192.168.206.122@tcp [ 808.815209] Key type .llcrypt unregistered [ 808.817789] Key type ._llcrypt unregistered [ 826.697633] Key type ._llcrypt registered [ 826.699630] Key type .llcrypt registered [ 826.769696] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 827.468961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.511282] alg: No test for adler32 (adler32-zlib) [ 828.562296] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 828.799665] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 830.441158] Key type lgssc registered [ 831.326933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.085297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.137634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.577573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.943990] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 849.888676] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.913740] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 849.926580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.326699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 854.332955] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 860.843212] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.798754] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.632617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.969521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.153816] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 882.881157] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.284504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 896.484118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 896.494640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.511365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.556846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 899.560162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.564756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.584821] Lustre: Skipped 1 previous similar message [ 900.795427] Lustre: server umount lustre-OST0000 complete [ 904.678992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.696040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.704699] Lustre: Skipped 1 previous similar message [ 909.794837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 909.801258] Lustre: Skipped 1 previous similar message [ 909.992387] Lustre: server umount lustre-MDT0000 complete [ 912.885490] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840966 with bad export cookie 13184354016390449097 [ 912.893755] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.898873] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 913.133236] Lustre: server umount lustre-MDT0001 complete [ 921.962486] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 924.614645] Key type lgssc unregistered [ 924.898774] LNet: 33743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.902168] LNetError: 33743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.914577] LNet: Removed LNI 192.168.206.122@tcp [ 925.740506] Key type .llcrypt unregistered [ 925.750933] Key type ._llcrypt unregistered [ 945.977172] Key type ._llcrypt registered [ 945.978856] Key type .llcrypt registered [ 946.107444] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 947.511790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 947.553438] alg: No test for adler32 (adler32-zlib) [ 948.507422] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 948.791772] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 950.615269] Key type lgssc registered [ 951.728678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.934220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.950160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.527963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.831408] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 969.460338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.746947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.998085] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 978.903852] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.102259] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 988.044824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.369621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.563932] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 999.544328] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1005.806928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1020.384798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.394472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.404144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.409116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.415389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.423804] Lustre: Skipped 1 previous similar message [ 1023.294478] Lustre: server umount lustre-OST0000 complete [ 1026.528462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.549024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.140578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1031.148858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.172169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.523681] Lustre: server umount lustre-MDT0000 complete [ 1035.666786] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841089 with bad export cookie 8097335680512767698 [ 1035.669753] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.687328] LustreError: 34988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1036.062361] Lustre: server umount lustre-MDT0001 complete [ 1045.676457] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1047.921652] Key type lgssc unregistered [ 1048.117524] LNet: 38562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.126729] LNetError: 38562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.167813] LNet: Removed LNI 192.168.206.122@tcp [ 1048.935661] Key type .llcrypt unregistered [ 1048.938364] Key type ._llcrypt unregistered [ 1068.049409] Key type ._llcrypt registered [ 1068.050634] Key type .llcrypt registered [ 1068.143117] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1069.107864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.157523] alg: No test for adler32 (adler32-zlib) [ 1070.152379] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 1070.321306] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1071.951488] Key type lgssc registered [ 1072.870821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1079.076428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1079.101712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.508126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.394881] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1091.040923] LustreError: 39953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.061422] LustreError: 39953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1092.546476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.893230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1096.856245] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1103.154489] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1108.300683] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.168028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.539260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.791926] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1126.827167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1128.573855] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.459431] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1137.127470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.133425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.142162] Lustre: Skipped 2 previous similar messages [ 1137.178195] Lustre: Skipped 2 previous similar messages [ 1141.546246] Lustre: server umount lustre-MDT0000 complete [ 1141.553744] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.445709] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841198 with bad export cookie 6235333873875199833 [ 1144.453714] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.457965] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1146.674941] LustreError: 42250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.692023] LustreError: 42250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1146.695963] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.22@tcp (stopping) [ 1147.360448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.411122] LustreError: 42250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.411190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1149.426975] LustreError: 42250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1149.441685] Lustre: Skipped 1 previous similar message [ 1150.839813] Lustre: server umount lustre-MDT0001 complete [ 1171.147182] Lustre: server umount lustre-OST0000 complete [ 1183.043616] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1185.253220] Key type lgssc unregistered [ 1185.573439] LNet: 43811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.582482] LNetError: 43811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.608474] LNet: Removed LNI 192.168.206.122@tcp [ 1186.335918] Key type .llcrypt unregistered [ 1186.339930] Key type ._llcrypt unregistered [ 1204.393908] Key type ._llcrypt registered [ 1204.396408] Key type .llcrypt registered [ 1204.483299] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1205.411662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.434805] alg: No test for adler32 (adler32-zlib) [ 1206.507968] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 1206.734357] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1208.471216] Key type lgssc registered [ 1209.431802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1216.052486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1216.089618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.625168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.647857] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1227.745841] LustreError: 45201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.778012] LustreError: 45201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1230.364293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.791673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.465722] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1243.205903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1247.483667] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1252.884338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1253.199357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.252244] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1262.480185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1266.497950] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1272.802160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1272.807644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.817685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1272.828950] Lustre: Skipped 1 previous similar message [ 1272.843018] Lustre: Skipped 1 previous similar message [ 1274.195842] Lustre: lustre-OST0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1277.931592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.937190] Lustre: Skipped 1 previous similar message [ 1278.704609] Lustre: server umount lustre-OST0000 complete [ 1295.170556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.185344] Lustre: Skipped 1 previous similar message [ 1295.198163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.206838] Lustre: Skipped 1 previous similar message [ 1300.449988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.467444] Lustre: Skipped 1 previous similar message [ 1301.476988] Lustre: server umount lustre-MDT0000 complete [ 1302.500703] LustreError: 45202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.673864] LustreError: 47044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841358 with bad export cookie 1954025500665557435 [ 1304.679479] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.689981] LustreError: 47044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1311.129542] Lustre: server umount lustre-MDT0001 complete [ 1320.997100] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1323.681919] Key type lgssc unregistered [ 1323.947298] LNet: 48909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.963775] LNetError: 48909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.979342] LNet: Removed LNI 192.168.206.122@tcp [ 1324.900445] Key type .llcrypt unregistered [ 1324.903379] Key type ._llcrypt unregistered [ 1348.906442] Key type ._llcrypt registered [ 1348.908631] Key type .llcrypt registered [ 1348.984420] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1350.107985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.133834] alg: No test for adler32 (adler32-zlib) [ 1351.126745] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 1351.329274] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1352.951151] Key type lgssc registered [ 1354.152360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.921979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.948809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.349269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.367636] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1372.647043] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.680480] LustreError: 50327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1375.332979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.832524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.125039] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1387.912448] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.146504] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1400.614172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.034150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.395706] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1408.190718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1414.690661] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.181376] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1427.145738] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1438.862887] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 0h24m50s [ 1440.191860] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1442.275765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.281052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.307252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.152722] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1443.157854] Lustre: Skipped 2 previous similar messages [ 1444.210167] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1444.217258] Lustre: Skipped 3 previous similar messages [ 1449.264981] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1455.411485] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 1455.421087] Lustre: Skipped 3 previous similar messages [ 1456.095101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1456.222385] Lustre: server umount lustre-MDT0000 complete [ 1459.258709] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1461.563279] LustreError: 50323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.285349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.388485] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.612346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.660367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1465.650377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1468.345275] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1471.688673] Lustre: DEBUG MARKER: Wait for df (36671) ... [ 1473.057491] Lustre: DEBUG MARKER: done [ 1477.088974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1477.097724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.110247] Lustre: Skipped 2 previous similar messages [ 1477.119787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.125411] Lustre: Skipped 3 previous similar messages [ 1480.162483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1480.167947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.587159] Lustre: server umount lustre-OST0000 complete [ 1486.304569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1486.307626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.335519] Lustre: Skipped 1 previous similar message [ 1491.797584] Lustre: server umount lustre-MDT0000 complete [ 1492.455399] LustreError: 51342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.474445] LustreError: 51342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1494.984996] LustreError: 50308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841548 with bad export cookie 16455344946640745016 [ 1494.995445] LustreError: 50308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1494.997865] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.265330] Lustre: server umount lustre-MDT0001 complete [ 1505.163838] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1507.362851] Key type lgssc unregistered [ 1507.652511] LNet: 55252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.660101] LNetError: 55252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.671545] LNet: Removed LNI 192.168.206.122@tcp [ 1508.336165] Key type .llcrypt unregistered [ 1508.338831] Key type ._llcrypt unregistered [ 1546.156995] Key type ._llcrypt registered [ 1546.162555] Key type .llcrypt registered [ 1546.266681] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1547.369356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.408805] alg: No test for adler32 (adler32-zlib) [ 1548.397846] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 1548.638925] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1550.361309] Key type lgssc registered [ 1551.538716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.713820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.736753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.912407] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1558.932530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1559.063139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.295945] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1568.886528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.941932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1568.963661] Lustre: 57456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1569.146609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.793878] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1578.909664] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1583.322869] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1588.702290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1588.859651] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1589.063104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.140881] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1598.573847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1600.155293] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1606.386569] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 0h27m38s [ 1607.832673] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1613.448153] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1615.443775] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1615.496850] LustreError: 57461:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1635.666760] Lustre: lustre-MDT0000: Client c13aeee2-b02b-4263-b3ad-abde76dad919 (at 192.168.206.22@tcp) reconnecting [ 1655.591105] LustreError: 57461:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1657.794758] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1660.074308] Lustre: DEBUG MARKER: done [ 1665.505357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.512799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1665.514021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.517108] Lustre: Skipped 1 previous similar message [ 1667.667370] Lustre: server umount lustre-OST0000 complete [ 1671.649510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.665574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.699463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.716544] Lustre: Skipped 1 previous similar message [ 1672.738129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.742846] Lustre: Skipped 2 previous similar messages [ 1676.771612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.775954] Lustre: Skipped 1 previous similar message [ 1677.679628] Lustre: server umount lustre-MDT0000 complete [ 1681.155083] LustreError: 60214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841735 with bad export cookie 6803490856659735478 [ 1681.162070] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.164662] LustreError: 60214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1681.552331] Lustre: server umount lustre-MDT0001 complete [ 1692.058192] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1694.498466] Key type lgssc unregistered [ 1694.743970] LNet: 61776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.761900] LNetError: 61776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.788149] LNet: Removed LNI 192.168.206.122@tcp [ 1695.597974] Key type .llcrypt unregistered [ 1695.600759] Key type ._llcrypt unregistered [ 1733.548803] Key type ._llcrypt registered [ 1733.552123] Key type .llcrypt registered [ 1733.658154] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 1734.414222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.452425] alg: No test for adler32 (adler32-zlib) [ 1735.429606] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 1735.635355] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 1737.271158] Key type lgssc registered [ 1738.072452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1740.609217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1740.628582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.898606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1741.915791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1742.048919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.003066] Lustre: Failing over lustre-MDT0000 [ 1743.255066] Lustre: server umount lustre-MDT0000 complete [ 1748.146777] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:30:00 (1788841800) [ 1753.478533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.900696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.592721] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1766.118297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.429553] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1766.535586] Lustre: 64547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1767.055777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1771.788755] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1781.960662] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1787.594431] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.523958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.691769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1793.842670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.322757] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1800.411714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1807.404411] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1813.396655] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1821.153727] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.158485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.173238] Lustre: Skipped 2 previous similar messages [ 1821.185899] Lustre: Skipped 2 previous similar messages [ 1825.181386] Lustre: server umount lustre-MDT0000 complete [ 1828.389106] LustreError: 65284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841882 with bad export cookie 5021664482416275030 [ 1828.394792] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.405786] LustreError: 65284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1828.669754] Lustre: server umount lustre-MDT0001 complete [ 1831.451212] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1835.816482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1843.099019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.776257] Lustre: 63472:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841885/real 1788841885] req@00000000cf875bea x1875736567783424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1847.791167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.826653] Lustre: 63473:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841887/real 1788841887] req@00000000e8646cf5 x1875736567783552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1852.959836] Lustre: 63473:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841890/real 1788841890] req@000000009779b0b9 x1875736567783680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1853.986899] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x45b0893da79e3739 [ 1853.995935] Lustre: MGC192.168.206.122@tcp: Connection restored to 0@lo (at 0@lo) [ 1854.304067] LustreError: 66397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.320210] LustreError: 66397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1854.405921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.513924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1855.391180] Lustre: 63472:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841892/real 1788841892] req@00000000587cfc15 x1875736567783936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1856.417209] LustreError: 66398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.446533] LustreError: 66397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.467799] LustreError: 66397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1858.531313] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1859.492683] Lustre: 63472:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841897/real 1788841897] req@000000001f6448f2 x1875736567784064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1862.624969] LustreError: 68324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.652502] LustreError: 68324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1866.990290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.361038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1871.179877] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1872.358758] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1876.453295] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1876.883449] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1882.306995] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1890.315514] Lustre: server umount lustre-MDT0000 complete [ 1891.808851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.812269] LustreError: 68322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.824141] Lustre: Skipped 2 previous similar messages [ 1891.858881] LustreError: 68322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1893.537882] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841947 with bad export cookie 5021664482416277305 [ 1893.551065] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.551926] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1893.847841] Lustre: server umount lustre-MDT0001 complete [ 1898.824958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1906.583996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.290135] Lustre: 63474:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841950/real 1788841950] req@000000004c791e80 x1875736567809024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1912.321305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.496654] LustreError: 63470:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000837d6ef0 x1875736567810176/t0(0) o250->MGC192.168.206.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1918.866253] LustreError: 69058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.944765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.009988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1922.846610] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1930.462444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.749861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1933.852828] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 1939.530685] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1943.923974] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1950.067503] Lustre: server umount lustre-MDT0000 complete [ 1951.202178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1951.211963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.242421] LustreError: 69058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.262948] LustreError: 69058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1953.669561] LustreError: 64541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842007 with bad export cookie 5021664482416278684 [ 1953.676648] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.682047] LustreError: 64541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1954.317478] Lustre: server umount lustre-MDT0001 complete [ 1961.004922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1964.796665] Lustre: DEBUG MARKER: files compared the same [ 1977.088691] Lustre: server umount lustre-OST0000 complete [ 1991.426257] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 1993.735727] Key type lgssc unregistered [ 1993.959195] LNet: 74276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.983033] LNetError: 74276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.001677] LNet: Removed LNI 192.168.206.122@tcp [ 1994.784799] Key type .llcrypt unregistered [ 1994.787406] Key type ._llcrypt unregistered [ 2016.853553] Key type ._llcrypt registered [ 2016.856223] Key type .llcrypt registered [ 2016.923820] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2017.666541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2017.705321] alg: No test for adler32 (adler32-zlib) [ 2018.747547] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2019.002028] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2020.713644] Key type lgssc registered [ 2021.571575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2027.779144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2027.790975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.123754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.982589] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2039.264274] LustreError: 75690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.276362] LustreError: 75690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2040.279539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.501841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.818049] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2049.687627] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.352529] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2060.138914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2060.389487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.950196] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug +malloc all [ 2071.818379] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2074.742210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2085.353790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.354311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2085.372708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2085.387929] Lustre: Skipped 1 previous similar message [ 2090.143706] Lustre: server umount lustre-OST0000 complete [ 2095.584716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.592671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.605390] Lustre: Skipped 2 previous similar messages [ 2100.703760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.712025] Lustre: Skipped 2 previous similar messages [ 2105.824404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.829408] Lustre: Skipped 1 previous similar message [ 2107.359391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2107.659781] Lustre: server umount lustre-MDT0000 complete [ 2110.949662] LustreError: 75689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.970912] LustreError: 75689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2111.717487] LustreError: 75672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842165 with bad export cookie 3468829671912181178 [ 2111.724017] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.730323] LustreError: 75672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2112.219317] Lustre: server umount lustre-MDT0001 complete [ 2122.343643] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2125.006797] Key type lgssc unregistered [ 2125.277532] LNet: 79241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.286635] LNetError: 79241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.302540] LNet: Removed LNI 192.168.206.122@tcp [ 2126.138373] Key type .llcrypt unregistered [ 2126.140643] Key type ._llcrypt unregistered [ 2149.371118] Key type ._llcrypt registered [ 2149.373233] Key type .llcrypt registered [ 2149.476206] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2150.125497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.159075] alg: No test for adler32 (adler32-zlib) [ 2151.153821] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2151.298976] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2152.961215] Key type lgssc registered [ 2153.956973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2160.700261] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2160.713592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.223356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2181.343837] LustreError: 80627:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.122@tcp: failed processing log, type 4: rc = -110 [ 2207.967196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.766290] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2218.973723] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2225.318310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.448455] Lustre: *** cfs_fail_loc=706, val=0*** [ 2225.785460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.866271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2229.821403] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2231.267884] LustreError: 80653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.276449] LustreError: 80653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2235.899991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.128388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2238.792617] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2244.086524] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2248.655609] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2251.235993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2251.249614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.259356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.746029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2251.755162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.771604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.615427] Lustre: server umount lustre-OST0000 complete [ 2261.983873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2261.988925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.999823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2262.010590] Lustre: Skipped 2 previous similar messages [ 2265.836193] Lustre: server umount lustre-MDT0000 complete [ 2268.129485] LustreError: 81716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.140318] LustreError: 81716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2268.768782] LustreError: 80635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842322 with bad export cookie 8514297420551950336 [ 2268.773146] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.781957] LustreError: 80635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2268.992356] Lustre: server umount lustre-MDT0001 complete [ 2277.322371] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2279.513665] Key type lgssc unregistered [ 2279.811795] LNet: 84238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.820900] LNetError: 84238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.842798] LNet: Removed LNI 192.168.206.122@tcp [ 2280.526846] Key type .llcrypt unregistered [ 2280.532797] Key type ._llcrypt unregistered [ 2301.597367] Key type ._llcrypt registered [ 2301.600323] Key type .llcrypt registered [ 2301.686428] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2302.249092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.349034] alg: No test for adler32 (adler32-zlib) [ 2303.374130] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2303.524257] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2305.167173] Key type lgssc registered [ 2306.086580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2311.960957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2311.978750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.044366] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2323.029744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.383341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2326.690807] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2328.547507] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.563180] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2333.342596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.608023] LustreError: 86315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.626106] LustreError: 86315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2333.629136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.627039] LustreError: 86320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.648389] LustreError: 86320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2338.898545] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2339.820575] LustreError: 86321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.836490] LustreError: 86321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2344.932094] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.944178] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2346.317757] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.332161] Lustre: MGS: Client 2f76fb4a-74c9-42cf-b023-9cd1cd56b17d (at 0@lo) reconnecting [ 2346.353843] LustreError: 86331:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2346.550586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.615178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2350.619716] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2366.953816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.964444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2366.998227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.025119] Lustre: Skipped 1 previous similar message [ 2369.956309] Lustre: server umount lustre-OST0000 complete [ 2377.186475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2377.187260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.198430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.234494] Lustre: Skipped 2 previous similar messages [ 2379.891615] Lustre: server umount lustre-MDT0000 complete [ 2382.310198] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.325588] LustreError: 86317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2383.176349] Lustre: server umount lustre-MDT0001 complete [ 2386.437394] Lustre: server umount MGS complete [ 2395.883468] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:40:48 (1788842448) [ 2424.656363] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 2433.464425] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2469.094571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.646719] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2482.823379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.096779] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2483.202870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2483.363730] Lustre: lustre-MDT0001: new disk, initializing [ 2483.449439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2483.488802] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2483.498864] Lustre: 93483:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2483.515467] LustreError: 93483:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2488.438986] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2499.672861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.942806] Lustre: lustre-OST0000: new disk, initializing [ 2499.946628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2499.960043] Lustre: 94251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2500.064239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.848284] Lustre: 94526:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000908a5131 x1875737163374080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788842576 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2507.153608] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2517.020063] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2523.722575] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.745168] Lustre: MGS: Client 16473d47-1225-4d5c-a4e5-0dd0585cd68d (at 0@lo) reconnecting [ 2523.797813] Lustre: 95300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2523.820579] LustreError: 93469:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2523.829155] LustreError: 93469:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2523.852551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2523.942433] Lustre: lustre-MDT0000: new disk, initializing [ 2524.042278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.071414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2524.085354] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2525.156616] Lustre: 95324:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000eac7faf0 x1875737163381504/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788842595 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2529.217890] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2530.277551] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2534.329931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2534.352913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2534.372543] Lustre: Skipped 1 previous similar message [ 2534.431430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2559.393102] LustreError: 95861:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000400:0x2:0x0].0x0 (00000000a81bc1c4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2559.459425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.463462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.468649] Lustre: Skipped 1 previous similar message [ 2559.477970] Lustre: Skipped 1 previous similar message [ 2564.577381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.593401] Lustre: Skipped 2 previous similar messages [ 2565.712788] Lustre: server umount lustre-OST0000 complete [ 2569.698533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2569.699303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.714675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.733384] Lustre: Skipped 1 previous similar message [ 2569.756120] Lustre: Skipped 1 previous similar message [ 2575.173920] Lustre: server umount lustre-MDT0000 complete [ 2585.423419] Lustre: server umount lustre-MDT0001 complete [ 2589.378620] Lustre: server umount MGS complete [ 2599.607272] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:44:12 (1788842652) [ 2618.225394] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2621.500671] Key type lgssc unregistered [ 2621.932383] LNet: 97819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.948836] LNetError: 97819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.990954] LNet: Removed LNI 192.168.206.122@tcp [ 2623.357719] Key type .llcrypt unregistered [ 2623.364602] Key type ._llcrypt unregistered [ 2638.382180] Key type ._llcrypt registered [ 2638.385817] Key type .llcrypt registered [ 2638.505578] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2639.523645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2639.602065] alg: No test for adler32 (adler32-zlib) [ 2640.700443] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2640.997777] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2642.711161] Key type lgssc registered [ 2643.672031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2650.691358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2650.727534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.201525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.930422] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2662.387839] LustreError: 99019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.405719] LustreError: 99019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2664.129670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.800534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2671.398526] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2680.659809] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.155733] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2692.843342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.378718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.525891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2699.504558] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2705.910098] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2718.179339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.188112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.197056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.226203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2720.232328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.235306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.332151] Lustre: server umount lustre-OST0000 complete [ 2730.471127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.482369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.488034] Lustre: Skipped 1 previous similar message [ 2730.499100] Lustre: Skipped 2 previous similar messages [ 2732.751397] Lustre: server umount lustre-MDT0000 complete [ 2735.595773] LustreError: 99020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.637024] LustreError: 100864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842791 with bad export cookie 11259501296618024807 [ 2737.647504] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.664754] LustreError: 100864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2738.309583] Lustre: server umount lustre-MDT0001 complete [ 2753.231258] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2756.335128] Key type lgssc unregistered [ 2756.625668] LNet: 102550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2756.629407] LNetError: 102550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2756.645200] LNet: Removed LNI 192.168.206.122@tcp [ 2757.401431] Key type .llcrypt unregistered [ 2757.405880] Key type ._llcrypt unregistered [ 2773.727476] Key type ._llcrypt registered [ 2773.733203] Key type .llcrypt registered [ 2773.893232] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2775.015694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2775.097233] alg: No test for adler32 (adler32-zlib) [ 2776.122286] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2776.409673] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2778.097679] Key type lgssc registered [ 2778.944050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2786.118540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2786.123813] LustreError: 103620:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2786.156964] LustreError: 103630:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2786.161861] LustreError: 103620:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 2786.168950] LustreError: 103620:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2786.173163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.179698] LustreError: 103620:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2787.796782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.336986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.672042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2799.874916] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2804.250403] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing lsmod [ 2806.438634] LustreError: 104836:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2806.439157] LustreError: 104835:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2806.445192] LustreError: 104836:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2806.450615] LustreError: 104836:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2806.457540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.458460] LustreError: 104836:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2806.773890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.360948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2811.366223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.380192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.875973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.880781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.889471] Lustre: Skipped 1 previous similar message [ 2815.445786] Lustre: server umount lustre-MDT0000 complete [ 2816.993715] LustreError: 103668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.016720] LustreError: 103668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2818.723440] LustreError: 103642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842872 with bad export cookie 13699502917211989324 [ 2818.737572] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.750632] Lustre: 103660:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2819.001263] Lustre: server umount lustre-MDT0001 complete [ 2831.539367] Lustre: server umount lustre-OST0000 complete [ 2840.717807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.099219] LustreError: 105898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.176355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.807689] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2846.182884] LustreError: 105899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.301724] LustreError: 105898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.214213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.496301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2860.167839] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2870.027925] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2877.930759] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2888.084873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.669433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.428507] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2902.082662] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2912.242495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2920.421330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.427217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.450969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.455504] Lustre: Skipped 1 previous similar message [ 2922.464620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.477094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.495228] Lustre: Skipped 1 previous similar message [ 2924.260549] Lustre: server umount lustre-OST0000 complete [ 2931.173221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2931.182296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.193448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2934.019187] Lustre: server umount lustre-MDT0000 complete [ 2937.686530] LustreError: 105878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842991 with bad export cookie 13699502917211990003 [ 2937.689716] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.700339] LustreError: 105878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2938.136468] Lustre: server umount lustre-MDT0001 complete [ 2948.342890] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 2950.813884] Key type lgssc unregistered [ 2951.156044] LNet: 109452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.165622] LNetError: 109452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.186289] LNet: Removed LNI 192.168.206.122@tcp [ 2952.033251] Key type .llcrypt unregistered [ 2952.039070] Key type ._llcrypt unregistered [ 2972.458458] Key type ._llcrypt registered [ 2972.460429] Key type .llcrypt registered [ 2972.537586] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 2973.392272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2973.417420] alg: No test for adler32 (adler32-zlib) [ 2974.377344] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2974.543283] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 2976.167181] Key type lgssc registered [ 2977.237979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.425356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.440540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.828734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.512809] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 2996.222720] LustreError: 110844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.260656] LustreError: 110844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3001.313842] LustreError: 110845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.757517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.304603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.532189] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3018.177866] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3023.724784] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3030.733152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.017329] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3031.111405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.189425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3037.128870] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3044.162161] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3048.854832] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3052.393103] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3058.145418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3058.151988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.162977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3058.183015] Lustre: Skipped 1 previous similar message [ 3058.589314] Lustre: server umount lustre-OST0000 complete [ 3063.266917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.273591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.284344] Lustre: Skipped 1 previous similar message [ 3063.305624] Lustre: Skipped 2 previous similar messages [ 3068.389056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.396140] Lustre: Skipped 1 previous similar message [ 3068.560108] Lustre: server umount lustre-MDT0000 complete [ 3071.958714] LustreError: 110826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843125 with bad export cookie 9000895618363789246 [ 3071.962457] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.968259] LustreError: 110826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3072.332793] Lustre: server umount lustre-MDT0001 complete [ 3083.664660] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3086.039298] Key type lgssc unregistered [ 3086.335827] LNet: 114485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3086.346768] LNetError: 114485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3086.366351] LNet: Removed LNI 192.168.206.122@tcp [ 3087.205423] Key type .llcrypt unregistered [ 3087.207660] Key type ._llcrypt unregistered [ 3102.147396] Key type ._llcrypt registered [ 3102.149467] Key type .llcrypt registered [ 3102.231573] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3103.155834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3103.164126] alg: No test for adler32 (adler32-zlib) [ 3104.505419] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 3104.841341] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3106.599168] Key type lgssc registered [ 3108.236336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3117.632853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3117.648564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.308382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.183253] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3129.829782] LustreError: 115685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.847179] LustreError: 115685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3133.482676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.048286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.525294] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3146.982500] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3152.410564] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3158.935652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.250877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.415227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3165.123858] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3171.327882] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.712468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3188.720944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.738069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.220504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.221025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.243418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.249171] Lustre: Skipped 1 previous similar message [ 3193.731638] Lustre: server umount lustre-OST0000 complete [ 3199.458435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.459747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.470396] Lustre: Skipped 1 previous similar message [ 3199.485827] Lustre: Skipped 1 previous similar message [ 3203.723148] Lustre: server umount lustre-MDT0000 complete [ 3204.577286] LustreError: 115682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.600612] LustreError: 115682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3207.245433] LustreError: 115666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843261 with bad export cookie 14275729916066346180 [ 3207.252422] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3207.252579] LustreError: 115666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3207.619735] Lustre: server umount lustre-MDT0001 complete [ 3218.721489] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3221.471978] Key type lgssc unregistered [ 3221.820523] LNet: 119237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3221.833974] LNetError: 119237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3221.859382] LNet: Removed LNI 192.168.206.122@tcp [ 3222.838920] Key type .llcrypt unregistered [ 3222.842411] Key type ._llcrypt unregistered [ 3248.782870] Key type ._llcrypt registered [ 3248.785259] Key type .llcrypt registered [ 3248.855399] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3251.280170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3251.349996] alg: No test for adler32 (adler32-zlib) [ 3252.759926] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 3253.190889] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3254.959515] Key type lgssc registered [ 3256.914898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3265.407623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3265.422215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.902734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.469562] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3276.261322] LustreError: 120651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.274782] LustreError: 120651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3280.386459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.792410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3285.092853] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3292.410390] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3297.590774] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3303.583611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.896878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.332825] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3311.094803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3317.221791] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3324.297369] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3327.529615] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3327.540127] Lustre: Skipped 2 previous similar messages [ 3330.113887] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3330.142066] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3331.230685] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3331.236805] Lustre: Skipped 2 previous similar messages [ 3339.881432] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3339.888793] Lustre: Skipped 1 previous similar message [ 3345.173282] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3345.184371] Lustre: Skipped 2 previous similar messages [ 3346.353490] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.22@tcp 192.168.0.[2,10]@tcp [ 3351.224916] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.22/24@tcp in log lustre-MDT0000 [ 3351.235852] Lustre: Skipped 2 previous similar messages [ 3355.685889] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.22/24@tcp [ 3355.694582] Lustre: Skipped 1 previous similar message [ 3363.827528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3363.834987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.841196] LustreError: Skipped 3 previous similar messages [ 3363.854734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3367.393947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3367.395897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3367.396308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.403723] Lustre: Skipped 1 previous similar message [ 3368.899411] Lustre: server umount lustre-OST0000 complete [ 3372.514759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.519519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.528887] Lustre: Skipped 1 previous similar message [ 3377.642634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.648577] Lustre: Skipped 2 previous similar messages [ 3378.655816] Lustre: server umount lustre-MDT0000 complete [ 3381.799668] LustreError: 120631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843435 with bad export cookie 5086539125257239744 [ 3381.802982] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.811348] LustreError: 120631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3382.028483] Lustre: server umount lustre-MDT0001 complete [ 3391.727676] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3394.245409] Key type lgssc unregistered [ 3394.543506] LNet: 125642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.561201] LNetError: 125642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3395.623674] LNet: Removed LNI 192.168.206.122@tcp [ 3396.470869] Key type .llcrypt unregistered [ 3396.472562] Key type ._llcrypt unregistered [ 3427.538278] Key type ._llcrypt registered [ 3427.544756] Key type .llcrypt registered [ 3427.716653] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3429.208240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.220696] alg: No test for adler32 (adler32-zlib) [ 3430.365443] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 3430.652067] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3432.343159] Key type lgssc registered [ 3433.667801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.418361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3441.431297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.788147] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3442.808848] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.22/24@tcp [ 3442.833203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.339473] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3452.407453] LustreError: 127201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.425219] LustreError: 127201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3455.667458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.930875] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3455.962391] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.22/24@tcp [ 3456.015505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.398866] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3467.662765] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3473.529099] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3480.108185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.429641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.817225] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3487.606693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3492.758451] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3503.074071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.076143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.094569] Lustre: Skipped 1 previous similar message [ 3503.111383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.196371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.203283] Lustre: Skipped 2 previous similar messages [ 3509.127844] Lustre: server umount lustre-OST0000 complete [ 3513.313414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.316828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.337129] Lustre: Skipped 1 previous similar message [ 3513.355698] Lustre: Skipped 1 previous similar message [ 3518.433179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.442684] Lustre: Skipped 1 previous similar message [ 3519.180856] Lustre: server umount lustre-MDT0000 complete [ 3522.962277] LustreError: 127183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843576 with bad export cookie 4128970304988042763 [ 3522.968031] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3522.975950] LustreError: 127183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3523.496923] Lustre: server umount lustre-MDT0001 complete [ 3535.849710] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3539.543288] Key type lgssc unregistered [ 3539.817435] LNet: 130787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.836795] LNetError: 130787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.863376] LNet: Removed LNI 192.168.206.122@tcp [ 3541.091916] Key type .llcrypt unregistered [ 3541.096362] Key type ._llcrypt unregistered [ 3569.150646] Key type ._llcrypt registered [ 3569.154652] Key type .llcrypt registered [ 3569.274643] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 3584.075607] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3585.166820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3585.218665] alg: No test for adler32 (adler32-zlib) [ 3586.224713] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 3586.417720] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3588.095166] Key type lgssc registered [ 3588.950914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3620.084379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3620.148768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.319667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3621.361199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3621.456210] Lustre: lustre-MDT0000: new disk, initializing [ 3621.524276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.545708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3625.323225] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3635.704131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.806573] Lustre: 134197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3635.860461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3635.864411] Lustre: Skipped 1 previous similar message [ 3635.929983] Lustre: lustre-MDT0001: new disk, initializing [ 3636.004325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.040021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3636.049180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3639.456783] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3647.397695] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3652.789636] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3658.715181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3658.945773] Lustre: lustre-OST0000: new disk, initializing [ 3658.956302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3658.964879] Lustre: 136163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3659.056849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.318566] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3666.498667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3666.510974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3666.669085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3673.464309] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3678.107883] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3678.321427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3682.852260] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3683.023895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3687.395328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.400049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.412966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3687.417429] Lustre: Skipped 1 previous similar message [ 3691.258281] Lustre: server umount lustre-OST0000 complete [ 3697.637994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.642974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3697.651517] Lustre: Skipped 1 previous similar message [ 3697.656936] LustreError: Skipped 1 previous similar message [ 3697.658155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3697.667932] Lustre: Skipped 1 previous similar message [ 3701.188088] Lustre: server umount lustre-MDT0000 complete [ 3704.263163] LustreError: 134189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843758 with bad export cookie 824933475290937309 [ 3704.267664] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3704.270461] LustreError: 134189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3704.545572] Lustre: server umount lustre-MDT0001 complete [ 3712.462877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.891154] LustreError: 138381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.920419] LustreError: 138381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3712.993597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.912567] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3718.114161] LustreError: 138382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.212873] LustreError: 138381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.332995] LustreError: 138382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.487680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.865851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.160591] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3740.517759] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.357370] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3750.997853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3751.331390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.288501] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3763.478864] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3766.995140] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3767.157236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3770.931786] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3771.157276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3779.545018] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.731828] Lustre: lustre-OST0001: new disk, initializing [ 3779.735723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3779.741295] Lustre: 141751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3779.803898] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.789392] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3787.354850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3787.366264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3787.485747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3791.916716] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3792.067279] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3795.798680] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3795.971616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3800.346924] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99dac7a37800.ost_server_uuid 50 [ 3801.421485] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99dac7a37800.ost_server_uuid in FULL state after 0 sec [ 3808.229387] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3808.235538] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.237063] LustreError: Skipped 1 previous similar message [ 3808.246178] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3808.252843] Lustre: Skipped 1 previous similar message [ 3808.275048] Lustre: Skipped 1 previous similar message [ 3813.353684] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3813.358183] Lustre: Skipped 1 previous similar message [ 3813.898621] Lustre: server umount lustre-OST0001 complete [ 3818.469102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.471951] LustreError: 140202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.474527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3818.487054] Lustre: Skipped 1 previous similar message [ 3818.497821] LustreError: 140202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3823.384453] Lustre: server umount lustre-OST0000 complete [ 3828.708043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.721992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.727180] Lustre: Skipped 2 previous similar messages [ 3833.160274] Lustre: server umount lustre-MDT0000 complete [ 3833.829165] LustreError: 139345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.856458] LustreError: 139345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3836.606426] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843890 with bad export cookie 824933475290938191 [ 3836.616205] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3836.619255] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.910191] Lustre: server umount lustre-MDT0001 complete [ 3856.826279] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 3859.344696] Key type lgssc unregistered [ 3859.592505] LNet: 144750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3859.598087] LNetError: 144750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3859.620696] LNet: Removed LNI 192.168.206.122@tcp [ 3860.381460] Key type .llcrypt unregistered [ 3860.386787] Key type ._llcrypt unregistered [ 3897.419398] Key type ._llcrypt registered [ 3897.423392] Key type .llcrypt registered [ 3897.531081] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 3909.728291] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 3910.754560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3910.785225] alg: No test for adler32 (adler32-zlib) [ 3911.866885] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 3912.117275] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 3913.847177] Key type lgssc registered [ 3914.745270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3942.950511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3942.974345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.183351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3944.218654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3944.305699] Lustre: lustre-MDT0000: new disk, initializing [ 3944.395481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.415017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3948.149085] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3957.756504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.876674] Lustre: 148882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.915602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3957.921840] Lustre: Skipped 1 previous similar message [ 3957.993628] Lustre: lustre-MDT0001: new disk, initializing [ 3958.092583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.121442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3958.134474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3962.077723] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3971.205881] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3975.629663] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3981.302054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.509409] Lustre: lustre-OST0000: new disk, initializing [ 3981.512984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3981.518681] Lustre: 150849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3981.591510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.696414] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 3991.628981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3991.644853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3991.751905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3994.936146] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3999.945756] Lustre: Failing over lustre-OST0000 [ 4000.129867] Lustre: server umount lustre-OST0000 complete [ 4017.631627] Lustre: 147105:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844055/real 1788844055] req@0000000074762a29 x1875738850524160/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788844071 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4017.671496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.662552] Lustre: 147103:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844056/real 1788844056] req@00000000914349bf x1875738850525056/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788844072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4018.720237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.789421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.284963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.311403] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4023.328198] Lustre: 147105:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844061/real 1788844061] req@00000000115d79be x1875738850526208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788844077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4023.362382] Lustre: 147105:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4024.119407] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4025.247624] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4028.153886] Lustre: Failing over lustre-MDT0000 [ 4028.175119] LustreError: 147102:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4028.406456] Lustre: server umount lustre-MDT0000 complete [ 4029.216529] Lustre: 147106:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844066/real 1788844066] req@00000000fe80d900 x1875738850527360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788844082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4029.229792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.239048] LustreError: 150045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.239059] LustreError: 150045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4029.265804] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4029.265851] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4029.270016] Lustre: Skipped 1 previous similar message [ 4033.311311] Lustre: 147104:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844071/real 1788844071] req@00000000ede77e63 x1875738850528512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788844087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4034.356499] LustreError: 148890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.369687] LustreError: 148890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4039.474368] LustreError: 148888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.488925] LustreError: 148888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4044.257759] Lustre: 148886:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844082/real 1788844082] req@0000000050e9a724 x1875738850537344/t0(0) o101->MGC192.168.206.122@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788844098 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4044.285064] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.303181] Lustre: 148886:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4044.313391] LustreError: 148888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.330856] LustreError: 148888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4045.727418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.376598] LustreError: 148894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.386162] LustreError: 148894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4054.662209] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4054.712238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.768649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4054.842065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4057.956076] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4060.146323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4060.164716] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4060.201805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4065.279023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.288476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.302519] Lustre: Skipped 1 previous similar message [ 4065.310470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.424900] Lustre: server umount lustre-OST0000 complete [ 4070.370550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.372381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4070.375136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.375143] Lustre: Skipped 1 previous similar message [ 4070.381728] Lustre: Skipped 2 previous similar messages [ 4075.494525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4075.504812] Lustre: Skipped 2 previous similar messages [ 4076.320100] Lustre: server umount lustre-MDT0000 complete [ 4079.709390] LustreError: 148876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844133 with bad export cookie 5331884409704505844 [ 4079.711936] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.728058] LustreError: 148876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4080.040260] Lustre: server umount lustre-MDT0001 complete [ 4090.364397] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 4092.781985] Key type lgssc unregistered [ 4093.045660] LNet: 154504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4093.051358] LNetError: 154504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4094.125098] LNet: Removed LNI 192.168.206.122@tcp [ 4094.829798] Key type .llcrypt unregistered [ 4094.832746] Key type ._llcrypt unregistered [ 4127.825582] Key type ._llcrypt registered [ 4127.828522] Key type .llcrypt registered [ 4127.932555] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4128.739467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.781048] alg: No test for adler32 (adler32-zlib) [ 4129.889222] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 4130.162725] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 4131.857276] Key type lgssc registered [ 4132.785498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4139.346895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4139.372410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.829806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.612415] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4150.243490] LustreError: 156017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.259421] LustreError: 156017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4151.726689] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4155.060973] LustreError: 156046:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4155.075160] LustreError: 156046:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4155.085840] LustreError: 156046:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4155.355415] Lustre: server umount lustre-MDT0000 complete [ 4167.455754] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4175.050685] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4209.003105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.231386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.261697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.320890] Lustre: lustre-MDT0000: new disk, initializing [ 4209.424211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.444964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4213.857664] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4225.676957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.817249] Lustre: 159350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4225.968186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4225.979296] Lustre: Skipped 1 previous similar message [ 4226.068878] Lustre: lustre-MDT0001: new disk, initializing [ 4226.176878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.214887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4226.235295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4230.950571] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4239.282903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4244.697992] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4252.056228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.313738] Lustre: lustre-OST0000: new disk, initializing [ 4252.319969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4252.325386] Lustre: 161317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.434571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.179551] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4261.986925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4262.007912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4262.104076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4269.822740] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4274.452191] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4276.091549] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:12:08 (1788844328) [ 4512.750706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.765112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.802855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.267428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.283833] Lustre: Skipped 1 previous similar message [ 4518.283373] Lustre: server umount lustre-MDT0000 complete [ 4518.394140] LustreError: 163096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.410409] LustreError: 163096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4522.366660] LustreError: 163088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844576 with bad export cookie 9592792855329958517 [ 4522.367599] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.379894] LustreError: 163088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4523.489265] LustreError: 159358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.489778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.565962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4523.571405] Lustre: Skipped 3 previous similar messages [ 4527.519613] LustreError: 161776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.520761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4528.735642] Lustre: server umount lustre-MDT0001 complete [ 4538.541937] Lustre: server umount lustre-OST0000 complete [ 4546.512318] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4555.000158] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4591.541199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.809701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4591.856356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4591.972905] Lustre: lustre-MDT0000: new disk, initializing [ 4592.040388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.052199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4596.559837] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4606.766832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.896556] Lustre: 165984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4606.950473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4606.953909] Lustre: Skipped 1 previous similar message [ 4607.024324] Lustre: lustre-MDT0001: new disk, initializing [ 4607.113498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4607.140984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4607.151426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4611.454779] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4620.897612] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.621309] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4631.626295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4631.916383] Lustre: lustre-OST0000: new disk, initializing [ 4631.920259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4631.934431] Lustre: 167952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4632.045647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.112623] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4641.348519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4641.362667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4641.520348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4647.392283] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4652.134203] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4652.433877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4656.984135] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4657.168614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4661.221520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.226194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4661.232109] Lustre: Skipped 1 previous similar message [ 4666.345868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4666.351959] Lustre: Skipped 1 previous similar message [ 4672.991210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4673.147108] Lustre: server umount lustre-OST0000 complete [ 4676.577881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.601848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.608127] Lustre: Skipped 4 previous similar messages [ 4682.539535] Lustre: server umount lustre-MDT0000 complete [ 4686.383348] LustreError: 165975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844740 with bad export cookie 9592792855330117347 [ 4686.385326] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.393220] LustreError: 165975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4686.896760] Lustre: server umount lustre-MDT0001 complete [ 4695.620602] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:19:07 (1788844747) [ 4710.276468] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4718.968626] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4749.965819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.179350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4750.206546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4750.300382] Lustre: lustre-MDT0000: new disk, initializing [ 4750.363863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.381693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4754.257388] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4763.519426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.642427] Lustre: 172708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4763.934055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4768.095581] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4777.700553] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4783.374854] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4789.720818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4789.960857] Lustre: lustre-OST0000: new disk, initializing [ 4789.963174] Lustre: Skipped 1 previous similar message [ 4789.969331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4789.975382] Lustre: Skipped 2 previous similar messages [ 4789.980755] Lustre: 174686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4795.811916] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4799.056397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4799.064742] Lustre: Skipped 1 previous similar message [ 4799.071681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4799.190091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4805.104431] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4814.827820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.829454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4814.831717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.831726] Lustre: Skipped 2 previous similar messages [ 4814.842603] Lustre: Skipped 2 previous similar messages [ 4814.880130] LustreError: Skipped 1 previous similar message [ 4817.513037] Lustre: server umount lustre-OST0000 complete [ 4827.020055] Lustre: server umount lustre-MDT0000 complete [ 4830.179079] LustreError: 174183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.204449] LustreError: 174183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4830.583426] LustreError: 174685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844884 with bad export cookie 9592792855330118229 [ 4830.588337] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.593040] LustreError: 174685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4831.207615] Lustre: server umount lustre-MDT0001 complete [ 4839.510548] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:21:31 (1788844891) [ 4854.877451] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 4864.916918] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 4899.467394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4899.697786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4899.744839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4899.824475] Lustre: lustre-MDT0000: new disk, initializing [ 4899.906730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.913905] Lustre: Skipped 2 previous similar messages [ 4899.928787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4904.402924] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4915.141632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.309830] Lustre: 179005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4915.652637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4919.793906] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4928.977328] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4935.323884] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4943.392295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4943.800551] Lustre: 180985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4951.248516] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 4952.668612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4952.845744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4961.331900] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4973.536160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4973.543880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.558505] Lustre: Skipped 2 previous similar messages [ 4973.563448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4973.570574] Lustre: Skipped 4 previous similar messages [ 4974.986922] Lustre: server umount lustre-OST0000 complete [ 4982.240474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4982.251940] LustreError: Skipped 1 previous similar message [ 4985.053424] Lustre: server umount lustre-MDT0000 complete [ 4988.376245] LustreError: 178997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845042 with bad export cookie 9592792855330119314 [ 4988.380427] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4988.391601] LustreError: 178997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4988.689818] Lustre: server umount lustre-MDT0001 complete [ 4999.858328] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 5002.443425] Key type lgssc unregistered [ 5002.712173] LNet: 182850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5002.721518] LNetError: 182850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5002.750386] LNet: Removed LNI 192.168.206.122@tcp [ 5003.545455] Key type .llcrypt unregistered [ 5003.549505] Key type ._llcrypt unregistered [ 5026.672233] Key type ._llcrypt registered [ 5026.674354] Key type .llcrypt registered [ 5026.756831] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5027.806799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5027.850441] alg: No test for adler32 (adler32-zlib) [ 5028.949715] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 5029.207360] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 5030.895196] Key type lgssc registered [ 5032.128373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5039.589248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5039.623654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5041.011570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.555644] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5050.343862] LustreError: 184263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.354868] LustreError: 184263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5053.498151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.806450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.493426] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5063.537457] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5068.553991] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5074.948334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.349512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.032345] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5085.637352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5087.811708] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5104.097984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5104.108192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.116797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.146991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.147146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.147261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5106.167883] Lustre: Skipped 1 previous similar message [ 5109.900660] Lustre: server umount lustre-OST0000 complete [ 5115.364060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5115.369716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.380316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.359945] Lustre: server umount lustre-MDT0000 complete [ 5121.509427] LustreError: 185348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.530182] LustreError: 185348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5122.640857] LustreError: 184244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845176 with bad export cookie 16252446578873934843 [ 5122.642633] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.655144] LustreError: 184244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5122.922913] Lustre: server umount lustre-MDT0001 complete [ 5132.590758] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 5136.039037] Key type lgssc unregistered [ 5136.446435] LNet: 187815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.450557] LNetError: 187815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5136.480144] LNet: Removed LNI 192.168.206.122@tcp [ 5137.339761] Key type .llcrypt unregistered [ 5137.342590] Key type ._llcrypt unregistered [ 5163.055359] Key type ._llcrypt registered [ 5163.057826] Key type .llcrypt registered [ 5163.184704] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5164.476981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5164.540732] alg: No test for adler32 (adler32-zlib) [ 5165.525370] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 5165.784618] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 5167.480179] Key type lgssc registered [ 5168.263816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5175.741761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5175.759780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.141748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.615768] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5187.557350] LustreError: 189227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.596492] LustreError: 189227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5190.144803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.740794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.697490] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5201.103953] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5206.030802] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5211.885632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5212.162115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.402935] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5222.457268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5224.987869] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5230.562785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5230.568779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.601634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5231.945984] Lustre: lustre-OST0000: Not available for connect from 192.168.206.22@tcp (stopping) [ 5232.612555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5232.630861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.651990] Lustre: server umount lustre-OST0000 complete [ 5243.460452] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99dac7d00800.ost_server_uuid 50 [ 5244.869879] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5262.309502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5262.311569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.326452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.355563] Lustre: Skipped 1 previous similar message [ 5262.375796] Lustre: Skipped 3 previous similar messages [ 5264.676369] Lustre: server umount lustre-MDT0000 complete [ 5267.426462] LustreError: 189227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.438778] LustreError: 189227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5269.050507] LustreError: 189208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845322 with bad export cookie 1664737134445641977 [ 5269.069386] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.095900] LustreError: 189208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5269.605105] Lustre: server umount lustre-MDT0001 complete [ 5278.559541] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:28:50 (1788845330) [ 5284.249289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.548190] LustreError: 193141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.610772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.797787] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5289.952883] LustreError: 193142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.091213] LustreError: 193141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.387882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.779527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5301.178854] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5307.876952] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5313.876960] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5320.156361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5320.519690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5325.871960] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5330.735898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5332.120487] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.218054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5337.425576] Lustre: lustre-OST0001: new disk, initializing [ 5337.429344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5337.435234] Lustre: 195983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5337.533202] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.886012] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5343.821885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5343.835530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5343.966971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5350.538837] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5354.471596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.472386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.487515] Lustre: Skipped 1 previous similar message [ 5358.842605] Lustre: server umount lustre-OST0000 complete [ 5359.596295] LustreError: 196654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.613818] LustreError: 196654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5363.629503] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5363.814896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5367.278553] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5367.446474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5376.821714] LustreError: 194965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.842455] LustreError: 194965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5380.064286] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5380.070912] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.076853] LustreError: Skipped 1 previous similar message [ 5380.077848] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5380.077853] Lustre: Skipped 2 previous similar messages [ 5380.104220] Lustre: Skipped 1 previous similar message [ 5385.881792] Lustre: server umount lustre-OST0001 complete [ 5388.770787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5388.782055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.793082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.798068] Lustre: Skipped 3 previous similar messages [ 5394.806902] Lustre: server umount lustre-MDT0000 complete [ 5397.710346] LustreError: 193121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845451 with bad export cookie 1664737134445643048 [ 5397.713615] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5397.718638] LustreError: 193121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5397.944338] Lustre: server umount lustre-MDT0001 complete [ 5415.963301] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:31:08 (1788845468) [ 5421.429545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.599482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5421.623641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5421.759053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.129550] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5432.614369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.704221] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5432.742679] Lustre: 199419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5436.516645] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5442.919249] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5447.353535] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5452.637946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5452.797811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5458.786391] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5460.087883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5464.827822] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.627847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5470.685091] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5476.367167] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5479.075477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5483.850880] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5489.648126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.652863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5489.669457] Lustre: Skipped 2 previous similar messages [ 5489.676173] Lustre: Skipped 2 previous similar messages [ 5492.568414] Lustre: server umount lustre-OST0000 complete [ 5494.757422] LustreError: 201267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.772582] LustreError: 201267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5509.601252] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5514.231403] Lustre: server umount lustre-OST0001 complete [ 5524.370505] Lustre: server umount lustre-MDT0000 complete [ 5528.322487] LustreError: 201298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845582 with bad export cookie 1664737134445644441 [ 5528.326637] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.334222] LustreError: 201298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5528.675667] Lustre: server umount lustre-MDT0001 complete [ 5549.841587] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:33:22 (1788845602) [ 5564.519861] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 5572.549409] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 5605.257905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.478365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5605.509568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5605.676979] Lustre: lustre-MDT0000: new disk, initializing [ 5605.790375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.796701] Lustre: Skipped 3 previous similar messages [ 5605.810652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5611.580931] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5622.182628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.310809] Lustre: 207512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.362192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5622.366435] Lustre: Skipped 1 previous similar message [ 5622.458618] Lustre: lustre-MDT0001: new disk, initializing [ 5622.590393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5622.606811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5626.993509] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5636.498116] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5642.164645] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5648.731573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5648.954470] Lustre: lustre-OST0000: new disk, initializing [ 5648.957962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5648.969417] Lustre: 209477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5654.787376] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5657.176527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5657.192390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5657.268718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5663.528566] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5667.706579] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5667.867494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5672.288490] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5672.527764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5678.073575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.074860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.081664] LustreError: Skipped 2 previous similar messages [ 5678.083732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.083739] Lustre: Skipped 6 previous similar messages [ 5678.108288] Lustre: Skipped 5 previous similar messages [ 5680.519055] Lustre: server umount lustre-OST0000 complete [ 5690.234842] Lustre: server umount lustre-MDT0000 complete [ 5693.409060] LustreError: 207523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.423604] LustreError: 207523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5693.525232] LustreError: 207505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845747 with bad export cookie 1664737134445646233 [ 5693.539114] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.551030] LustreError: 207505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5693.930790] Lustre: server umount lustre-MDT0001 complete [ 5702.212690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.139377] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5716.556776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.001808] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5730.864178] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5735.695520] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5742.050943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.353636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.361900] Lustre: Skipped 4 previous similar messages [ 5748.365113] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5756.396927] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5761.857436] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5762.188955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5767.594166] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5767.834185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5773.286907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.288191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.288649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.288653] Lustre: Skipped 4 previous similar messages [ 5773.326363] Lustre: Skipped 3 previous similar messages [ 5773.334391] LustreError: Skipped 1 previous similar message [ 5776.158738] Lustre: server umount lustre-OST0000 complete [ 5783.880086] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5790.168645] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5819.186107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5825.165380] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5832.297755] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.035634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 5839.084031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.105475] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5839.109376] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.850937] Lustre: server umount lustre-OST0000 complete [ 5857.830680] Lustre: server umount lustre-MDT0000 complete [ 5859.811269] LustreError: 211692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.831871] LustreError: 211692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5860.175372] Lustre: 188539:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845898/real 1788845898] req@00000000badd4cb3 x1875740164600064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788845914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5861.277631] LustreError: 211677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845915 with bad export cookie 1664737134445647115 [ 5861.281293] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.289982] LustreError: 211677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5861.290644] Lustre: 188540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845898/real 1788845898] req@000000007a49431d x1875740164599168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788845914 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5861.621498] Lustre: server umount lustre-MDT0001 complete [ 5870.245796] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:38:42 (1788845922) [ 5876.546748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.440790] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5888.590120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.052897] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5899.198995] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5904.194779] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5909.656168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5916.982809] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5924.357471] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5928.908374] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5929.096912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5933.482960] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5933.716194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5939.884170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.068584] Lustre: lustre-OST0001: new disk, initializing [ 5940.073481] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5940.080050] Lustre: 221031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5945.984127] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 5948.487739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5948.499154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5948.586436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5953.151113] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5956.874914] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5957.072837] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5961.775056] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5962.014715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5964.265104] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.265238] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5964.266282] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5964.266289] Lustre: Skipped 5 previous similar messages [ 5964.273170] Lustre: Skipped 4 previous similar messages [ 5964.293460] LustreError: Skipped 1 previous similar message [ 5969.851948] Lustre: server umount lustre-OST0001 complete [ 5977.752138] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5978.078813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5982.809372] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5983.064358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6011.886633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.069524] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6012.082533] Lustre: Skipped 5 previous similar messages [ 6017.523398] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6017.532485] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6017.533846] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6017.554071] Lustre: Skipped 1 previous similar message [ 6017.822996] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6024.616793] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.070992] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6042.079174] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6042.270391] Lustre: server umount lustre-OST0001 complete [ 6073.328540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6073.462339] Lustre: server umount lustre-OST0000 complete [ 6083.812503] Lustre: server umount lustre-MDT0000 complete [ 6087.410210] LustreError: 218394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846141 with bad export cookie 1664737134445648081 [ 6087.424885] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.427444] LustreError: 218394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6094.053640] Lustre: server umount lustre-MDT0001 complete [ 6116.587800] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:42:49 (1788846169) [ 6122.438945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.669882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6122.714199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6127.533295] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6136.210886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.293444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6136.313317] Lustre: 226280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6140.976151] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6150.017395] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6156.468848] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6163.173912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6163.317402] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6169.459938] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6177.352653] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6185.675669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.768170] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6192.182516] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6199.136817] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6200.564684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6204.980319] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6205.184897] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6209.214312] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6209.443122] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6215.088288] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99dac6a59000.ost_server_uuid 50 [ 6216.481829] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99dac6a59000.ost_server_uuid in IDLE state after 0 sec [ 6217.898185] Lustre: Permanently deactivating lustre-OST0001 [ 6217.919498] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6221.331875] Lustre: Permanently reactivating lustre-OST0001 [ 6229.474142] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.489231] LustreError: Skipped 4 previous similar messages [ 6229.494870] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.526058] Lustre: Skipped 6 previous similar messages [ 6229.571742] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6229.584208] Lustre: Skipped 18 previous similar messages [ 6232.001110] Lustre: server umount lustre-OST0001 complete [ 6235.136922] LustreError: 228124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.156681] LustreError: 228124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 6243.294126] Lustre: server umount lustre-OST0000 complete [ 6254.438760] Lustre: server umount lustre-MDT0000 complete [ 6258.528031] LustreError: 228138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846312 with bad export cookie 1664737134445649369 [ 6258.536027] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6258.540623] LustreError: 228138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6259.095809] Lustre: server umount lustre-MDT0001 complete [ 6284.376619] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:45:36 (1788846336) [ 6290.983342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.248623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6291.295070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6291.304710] Lustre: Skipped 5 previous similar messages [ 6295.558853] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6303.282438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.364268] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6303.390475] Lustre: 232973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6307.555774] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6314.276788] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.979354] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6325.412164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6325.668442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6325.705917] Lustre: Permanently deactivating lustre-OST0000 [ 6325.718977] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6331.421092] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6331.425862] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6337.600658] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.158292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6344.227453] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6350.528135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6350.617031] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6358.115990] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.007617] Lustre: Permanently reactivating lustre-OST0000 [ 6360.022504] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6360.027695] Lustre: Skipped 2 previous similar messages [ 6368.831179] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6368.834472] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 000000004253066f, cur 1788846422 deadline 1788846415 last 1788846385 [ 6368.841388] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.862707] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6368.874086] Lustre: Skipped 1 previous similar message [ 6368.901525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6391.263175] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6391.398135] Lustre: server umount lustre-OST0001 complete [ 6401.502918] Lustre: server umount lustre-OST0000 complete [ 6411.236400] Lustre: server umount lustre-MDT0000 complete [ 6414.631593] LustreError: 232967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846468 with bad export cookie 1664737134445651049 [ 6414.633050] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.645582] LustreError: 232967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.939471] Lustre: server umount lustre-MDT0001 complete [ 6425.788740] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 6428.436158] Key type lgssc unregistered [ 6428.686989] LNet: 237863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6428.694180] LNetError: 237863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6428.723127] LNet: Removed LNI 192.168.206.122@tcp [ 6429.463930] Key type .llcrypt unregistered [ 6429.466864] Key type ._llcrypt unregistered [ 6450.274902] Key type ._llcrypt registered [ 6450.277892] Key type .llcrypt registered [ 6450.398826] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 6451.214959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6451.231362] alg: No test for adler32 (adler32-zlib) [ 6452.336956] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 6452.566455] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 6454.231145] Key type lgssc registered [ 6455.327200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6464.614903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6464.637283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.143213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.666416] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6476.259514] LustreError: 239346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.276460] LustreError: 239346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6479.005943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.170886] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6479.206736] Lustre: 239334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6479.219105] Lustre: Permanently deactivating lustre-MDT0001 [ 6479.232603] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6479.551752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6480.275740] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6480.288714] LustreError: 239377:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6480.308643] LustreError: 239377:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6480.319793] LustreError: 239377:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 6483.732654] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6490.684515] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6495.832706] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6502.084515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.513514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.558434] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.568596] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6508.151473] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6510.059305] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.088156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6515.205284] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6516.197413] LustreError: 241189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.223402] LustreError: 241189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6521.315717] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.341194] LustreError: 241323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6522.261959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.491943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6527.991938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6527.995820] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6528.926793] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6536.610615] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6540.451590] Lustre: Permanently reactivating lustre-MDT0001 [ 6540.466525] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6540.469718] Lustre: Skipped 2 previous similar messages [ 6546.796097] Lustre: lustre-MDT0001: Client fcf185e9-c01b-402e-9e6c-9fe7ae6a8476 (at 192.168.206.22@tcp) reconnecting [ 6553.043503] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6554.683313] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6559.868689] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6560.031517] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6564.207934] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6564.377681] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6566.961278] Lustre: Permanently deactivating lustre-MDT0001 [ 6566.986291] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6566.997295] Lustre: Skipped 2 previous similar messages [ 6575.712647] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6581.739738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6581.751848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.767398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.097435] Lustre: server umount lustre-MDT0000 complete [ 6584.294826] LustreError: 240090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.299091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.323728] Lustre: Skipped 1 previous similar message [ 6587.863339] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846641 with bad export cookie 14051561464176693945 [ 6587.869190] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6587.881894] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6588.289484] Lustre: server umount lustre-MDT0001 complete [ 6594.332873] Lustre: server umount lustre-OST0000 complete [ 6606.809131] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:50:58 (1788846658) [ 6617.448185] Lustre: server umount lustre-OST0001 complete [ 6625.328552] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 6640.358529] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 6676.971535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.164841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6677.170404] Lustre: Skipped 2 previous similar messages [ 6677.183871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6677.241278] Lustre: lustre-MDT0000: new disk, initializing [ 6677.306724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.333914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6681.958206] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6692.935152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.060057] Lustre: 247624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6693.113074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6693.119675] Lustre: Skipped 1 previous similar message [ 6693.202194] Lustre: lustre-MDT0001: new disk, initializing [ 6693.309229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6693.349606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6693.367824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6697.745257] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6706.229984] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6710.998864] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6716.924365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.178567] Lustre: lustre-OST0000: new disk, initializing [ 6717.182830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6717.193299] Lustre: 249592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6717.287417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.241413] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6726.205451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6726.218886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6726.329664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6730.406903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6732.706901] LustreError: 247632:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6741.602565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.782128] Lustre: lustre-OST0001: new disk, initializing [ 6741.785300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6741.796507] Lustre: 250704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6742.799135] LustreError: 247632:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6742.839173] LustreError: 250407:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6747.869754] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6748.730556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6748.739143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6748.859808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6752.839335] LustreError: 250407:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6756.667062] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6758.026356] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6761.319125] Lustre: lustre-OST0001: Not available for connect from 192.168.206.22@tcp (stopping) [ 6764.516244] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.520648] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.524596] LustreError: Skipped 1 previous similar message [ 6764.540365] Lustre: Skipped 1 previous similar message [ 6764.541491] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6764.551210] Lustre: Skipped 1 previous similar message [ 6769.634295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6769.641909] Lustre: Skipped 2 previous similar messages [ 6774.239180] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6774.383041] Lustre: server umount lustre-OST0001 complete [ 6774.769788] LustreError: 249575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.794719] LustreError: 249575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6793.185041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6793.194425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.214301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.220330] Lustre: Skipped 1 previous similar message [ 6793.250099] Lustre: Skipped 2 previous similar messages [ 6798.106379] Lustre: server umount lustre-OST0000 complete [ 6805.479427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.481925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6805.487285] Lustre: Skipped 1 previous similar message [ 6805.507250] Lustre: Skipped 3 previous similar messages [ 6808.858680] Lustre: server umount lustre-MDT0000 complete [ 6810.593036] LustreError: 247636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.619780] LustreError: 247636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6812.383199] LustreError: 249591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846866 with bad export cookie 14051561464176696983 [ 6812.392505] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.397366] LustreError: 249591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6818.746786] Lustre: server umount lustre-MDT0001 complete [ 6829.118673] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 6831.899893] Key type lgssc unregistered [ 6832.228278] LNet: 252737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6832.234547] LNetError: 252737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6832.254975] LNet: Removed LNI 192.168.206.122@tcp [ 6833.297530] Key type .llcrypt unregistered [ 6833.304069] Key type ._llcrypt unregistered [ 6869.943390] Key type ._llcrypt registered [ 6869.949522] Key type .llcrypt registered [ 6870.060708] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 6871.110684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6871.145287] alg: No test for adler32 (adler32-zlib) [ 6872.159646] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 6872.387667] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 6874.063181] Key type lgssc registered [ 6875.088314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6881.444841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6881.471280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.733663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6882.754768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6882.881405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.539522] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6894.656558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.730537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6894.762333] Lustre: 254942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6895.098166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6899.508257] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6906.200583] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6910.993315] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6916.764767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.899700] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6917.041795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.915250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6922.974942] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 6930.425576] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6984.162978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6984.169096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.179604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6984.675292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.690760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.064739] Lustre: server umount lustre-OST0000 complete [ 6992.956151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7003.925473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7004.319920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.789025] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7009.805103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7009.810993] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7009.904359] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7017.409228] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7060.962182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.966794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.990036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7062.489763] Lustre: server umount lustre-OST0000 complete [ 7066.086603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.090397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7066.096554] Lustre: Skipped 2 previous similar messages [ 7066.107262] Lustre: Skipped 1 previous similar message [ 7071.200858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.209466] Lustre: Skipped 2 previous similar messages [ 7072.252863] Lustre: server umount lustre-MDT0000 complete [ 7076.091288] LustreError: 254934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847130 with bad export cookie 12654969043824334545 [ 7076.092186] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.100346] LustreError: 254934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7076.637553] Lustre: server umount lustre-MDT0001 complete [ 7088.173609] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7092.131516] Key type lgssc unregistered [ 7092.592181] LNet: 260361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7092.605767] LNetError: 260361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.644906] LNet: Removed LNI 192.168.206.122@tcp [ 7093.896567] Key type .llcrypt unregistered [ 7093.898358] Key type ._llcrypt unregistered [ 7121.784316] Key type ._llcrypt registered [ 7121.786578] Key type .llcrypt registered [ 7122.002738] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7123.520104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7123.654714] alg: No test for adler32 (adler32-zlib) [ 7124.761485] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 7125.024365] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7126.703193] Key type lgssc registered [ 7127.927321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7134.601140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7134.621626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.079225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7140.535751] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7145.443512] LustreError: 261773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.459645] LustreError: 261773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7148.791765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.058842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7153.780433] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7161.274781] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7167.017648] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7174.036945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.352752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.867364] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7180.530646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7186.493888] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.731118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7221.750807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.780312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7224.805362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7224.805634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7224.805712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.831633] Lustre: Skipped 1 previous similar message [ 7226.647632] Lustre: server umount lustre-OST0000 complete [ 7230.945632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.952397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.965551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7235.042895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7235.043950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.166408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7240.178495] Lustre: Skipped 2 previous similar messages [ 7245.281204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7245.498279] Lustre: server umount lustre-MDT0000 complete [ 7248.892272] LustreError: 261753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847302 with bad export cookie 15279234667763294635 [ 7248.912252] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.260096] Lustre: server umount lustre-MDT0001 complete [ 7258.476072] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7261.037750] Key type lgssc unregistered [ 7261.302388] LNet: 266113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7261.312363] LNetError: 266113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7262.389217] LNet: Removed LNI 192.168.206.122@tcp [ 7263.079616] Key type .llcrypt unregistered [ 7263.083775] Key type ._llcrypt unregistered [ 7277.813055] Key type ._llcrypt registered [ 7277.816017] Key type .llcrypt registered [ 7277.953521] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7278.799867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7278.819499] alg: No test for adler32 (adler32-zlib) [ 7280.055493] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 7280.395752] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7282.151175] Key type lgssc registered [ 7283.727343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7291.530583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7291.547199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7292.978028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.686421] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7302.113854] LustreError: 267314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.129618] LustreError: 267314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7305.706883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.218915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.696567] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7317.533346] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7323.268709] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7329.647801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7329.898886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.856717] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7337.070671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7342.733425] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7357.920199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7357.923434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.934687] LustreError: Skipped 1 previous similar message [ 7357.936612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7357.953067] Lustre: Skipped 1 previous similar message [ 7357.973093] Lustre: Skipped 1 previous similar message [ 7359.570225] Lustre: server umount lustre-OST0000 complete [ 7367.653017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7367.663253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.685105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.462548] Lustre: server umount lustre-MDT0000 complete [ 7372.476564] LustreError: 270241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847426 with bad export cookie 3805063334638476221 [ 7372.482061] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.486785] LustreError: 270241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7372.705409] Lustre: server umount lustre-MDT0001 complete [ 7384.411571] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7387.004844] Key type lgssc unregistered [ 7387.303618] LNet: 270992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7387.321447] LNetError: 270992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7387.348881] LNet: Removed LNI 192.168.206.122@tcp [ 7388.224919] Key type .llcrypt unregistered [ 7388.227232] Key type ._llcrypt unregistered [ 7413.342372] Key type ._llcrypt registered [ 7413.344852] Key type .llcrypt registered [ 7413.439592] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7414.340871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7414.479435] alg: No test for adler32 (adler32-zlib) [ 7415.539919] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 7415.838841] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7417.551716] Key type lgssc registered [ 7418.913895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7427.448593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7427.465880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.882273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.350703] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7438.313986] LustreError: 272408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.336415] LustreError: 272408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7442.064216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.424299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7446.728336] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7454.171043] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.981722] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7466.217495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.590719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.045666] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7473.768584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7479.436740] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7503.840531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7503.857989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.867218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.871505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7504.876654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.879786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.420510] Lustre: server umount lustre-OST0000 complete [ 7513.056651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.060926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.076967] Lustre: Skipped 1 previous similar message [ 7513.106132] Lustre: Skipped 2 previous similar messages [ 7515.104152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.115741] Lustre: Skipped 1 previous similar message [ 7520.225241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.231860] Lustre: Skipped 1 previous similar message [ 7526.367142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.651141] Lustre: server umount lustre-MDT0000 complete [ 7529.528653] LustreError: 274644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.551720] LustreError: 274644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7530.736816] LustreError: 272389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847584 with bad export cookie 6069947942658873045 [ 7530.748837] LustreError: 272389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7530.749916] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.980977] Lustre: server umount lustre-MDT0001 complete [ 7543.202957] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7545.885929] Key type lgssc unregistered [ 7546.191659] LNet: 276710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7546.205608] LNetError: 276710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7546.233402] LNet: Removed LNI 192.168.206.122@tcp [ 7547.004588] Key type .llcrypt unregistered [ 7547.006696] Key type ._llcrypt unregistered [ 7563.976429] Key type ._llcrypt registered [ 7563.978435] Key type .llcrypt registered [ 7564.084529] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7565.103892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7565.249210] alg: No test for adler32 (adler32-zlib) [ 7566.394913] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 7566.623099] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7568.303246] Key type lgssc registered [ 7569.384648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7577.435519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7577.477732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7579.058206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.671736] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7588.326696] LustreError: 277945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.340382] LustreError: 277945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7592.520142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.836883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.257188] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7604.960344] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7610.376575] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7616.443712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.850154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.853146] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7624.062652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7629.906343] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7643.616137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7643.623719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.641600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.645431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.647383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.651871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.971713] Lustre: server umount lustre-OST0000 complete [ 7654.371254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7654.387766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.414396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.417062] Lustre: Skipped 1 previous similar message [ 7656.581286] Lustre: server umount lustre-MDT0000 complete [ 7660.011100] LustreError: 277927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.038412] LustreError: 277927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7660.808615] LustreError: 277912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847714 with bad export cookie 17102615546891086069 [ 7660.813441] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7660.827369] LustreError: 277912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7661.356088] Lustre: server umount lustre-MDT0001 complete [ 7672.862879] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 7675.284892] Key type lgssc unregistered [ 7675.574506] LNet: 281644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7675.586373] LNetError: 281644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7675.607218] LNet: Removed LNI 192.168.206.122@tcp [ 7676.622425] Key type .llcrypt unregistered [ 7676.626940] Key type ._llcrypt unregistered [ 7687.436500] Key type ._llcrypt registered [ 7687.441033] Key type .llcrypt registered [ 7687.588713] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7703.435165] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 7717.402489] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 7718.357296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7718.471144] alg: No test for adler32 (adler32-zlib) [ 7719.685489] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 7719.991682] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 7721.719231] Key type lgssc registered [ 7722.841070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7764.960445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7765.015497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.437074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7766.477675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7766.644757] Lustre: lustre-MDT0000: new disk, initializing [ 7766.832814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7766.870740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7773.064689] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7784.916749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7785.041341] Lustre: 285195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7785.140610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7785.147252] Lustre: Skipped 1 previous similar message [ 7785.272306] Lustre: lustre-MDT0001: new disk, initializing [ 7785.420739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7785.465929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7785.490669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7790.926792] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7800.581045] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7807.610378] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.513205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.901905] Lustre: lustre-OST0000: new disk, initializing [ 7815.908145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7815.917859] Lustre: 287163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7816.088366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.607693] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7823.987059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7824.010219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7824.229508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7834.795935] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7840.920319] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7841.098822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7846.982369] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7847.267372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7849.956351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7849.956800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.973613] LustreError: Skipped 1 previous similar message [ 7849.986398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7855.076597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7855.082569] Lustre: Skipped 2 previous similar messages [ 7860.206879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.216827] Lustre: Skipped 1 previous similar message [ 7864.287246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7864.676582] Lustre: server umount lustre-OST0000 complete [ 7870.442428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.458247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.471661] Lustre: Skipped 2 previous similar messages [ 7870.497966] Lustre: Skipped 1 previous similar message [ 7875.555784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.571383] Lustre: Skipped 1 previous similar message [ 7883.231154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7883.799752] Lustre: server umount lustre-MDT0000 complete [ 7888.590182] LustreError: 285188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847942 with bad export cookie 13004214011575475520 [ 7888.597891] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.613937] LustreError: 285188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7889.212796] Lustre: server umount lustre-MDT0001 complete [ 7899.448555] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:12:31 (1788847951) [ 7907.374627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.088813] LustreError: 289580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.107521] LustreError: 289580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7908.140380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7912.738603] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7913.453859] LustreError: 289581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.562223] LustreError: 289580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.663242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.059549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.633214] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7934.432584] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.909501] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7949.015959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7949.661909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7956.033458] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 7963.688851] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.489607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7979.501926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.528556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7979.537912] Lustre: Skipped 2 previous similar messages [ 7982.566743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7982.587582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.675809] Lustre: server umount lustre-OST0000 complete [ 7992.803189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.829847] Lustre: Skipped 1 previous similar message [ 7995.534817] Lustre: server umount lustre-MDT0000 complete [ 7997.923915] LustreError: 289575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.950870] LustreError: 289575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7999.555120] LustreError: 289561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848053 with bad export cookie 13004214011575476402 [ 7999.569342] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7999.571485] LustreError: 289561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8000.039500] Lustre: server umount lustre-MDT0001 complete [ 8012.943857] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8016.130551] Key type lgssc unregistered [ 8016.411581] LNet: 293137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8016.416558] LNetError: 293137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8016.429565] LNet: Removed LNI 192.168.206.122@tcp [ 8017.430077] Key type .llcrypt unregistered [ 8017.433715] Key type ._llcrypt unregistered [ 8046.237312] Key type ._llcrypt registered [ 8046.240818] Key type .llcrypt registered [ 8046.375834] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 8064.782485] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8065.748495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8065.962376] alg: No test for adler32 (adler32-zlib) [ 8067.239927] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 8067.604673] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8069.312638] Key type lgssc registered [ 8070.490799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.087804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.138713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.364147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8121.394179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8121.459198] Lustre: lustre-MDT0000: new disk, initializing [ 8121.535278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.559287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8127.038642] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8140.845604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.972220] Lustre: 296962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8141.021449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8141.027167] Lustre: Skipped 1 previous similar message [ 8141.080882] Lustre: lustre-MDT0001: new disk, initializing [ 8141.179387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.242925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8141.266995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8147.051271] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8158.026491] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8163.689418] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8172.939386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8173.464081] Lustre: lustre-OST2710: new disk, initializing [ 8173.471910] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8173.520705] Lustre: 298931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8173.631782] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8182.449132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8182.463967] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8182.679204] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8184.309457] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8196.978776] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8205.463092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.813301] Lustre: lustre-OST03e8: new disk, initializing [ 8205.821495] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8205.834811] Lustre: 299997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8206.010487] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8213.476446] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8215.197222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8215.209726] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8215.294351] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8222.534566] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8231.874582] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8232.173911] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8237.075384] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8237.327964] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8242.207503] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8242.405809] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8247.938635] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8248.134754] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8253.920487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8253.930441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.941452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.597825] Lustre: server umount lustre-MDT0000 complete [ 8256.497838] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8256.524963] Lustre: Skipped 2 previous similar messages [ 8258.611499] LustreError: 298930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848312 with bad export cookie 16178083873660532066 [ 8258.634574] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8259.111810] Lustre: server umount lustre-MDT0001 complete [ 8274.017300] Lustre: server umount lustre-OST2710 complete [ 8277.791197] Lustre: 294759:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788848315/real 1788848315] req@000000008f5a7924 x1875743207398144/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788848331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8277.835269] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.146844] Lustre: server umount lustre-OST03e8 complete [ 8296.524827] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 8308.726197] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8346.013897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.279178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8346.329595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8346.402271] Lustre: lustre-MDT0000: new disk, initializing [ 8346.476564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.500758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8350.933065] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8361.733578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.830814] Lustre: 305246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8361.883780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8361.897275] Lustre: Skipped 1 previous similar message [ 8361.987561] Lustre: lustre-MDT0001: new disk, initializing [ 8362.120723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8362.162371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8362.178495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8366.632251] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8376.391815] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.865763] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8388.761453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8389.008986] Lustre: lustre-OST0000: new disk, initializing [ 8389.011950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8389.017693] Lustre: 307215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8389.110375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.982171] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8398.393455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8398.404857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8398.568690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8404.124218] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8408.503290] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8408.684801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8412.909416] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8413.170767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8419.302026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8419.317308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.328594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.198514] Lustre: server umount lustre-OST0000 complete [ 8428.513525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8428.520705] LustreError: Skipped 1 previous similar message [ 8428.524811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.536713] Lustre: Skipped 1 previous similar message [ 8428.545321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8428.553055] Lustre: Skipped 1 previous similar message [ 8430.901418] Lustre: server umount lustre-MDT0000 complete [ 8434.114023] LustreError: 305238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848488 with bad export cookie 16178083873660533746 [ 8434.118022] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8434.122556] LustreError: 305238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8434.419670] Lustre: server umount lustre-MDT0001 complete [ 8442.460096] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:21:34 (1788848494) [ 8443.933537] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8445.614373] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:21:37 (1788848497) [ 8449.586434] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8481.328180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.631423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8481.659432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8481.954142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.400510] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8501.925353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.135574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8502.195719] Lustre: 311069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8507.919946] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8531.046843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8531.381155] Lustre: Denying initial registration attempt from nid 192.168.206.122@tcp, specified as failover [ 8531.396220] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8531.410430] LustreError: 312524:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8531.423602] LustreError: 312524:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8531.428895] LustreError: 312524:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8531.526542] Lustre: server umount lustre-OST0000 complete [ 8531.539070] LustreError: 312524:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8543.203036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8543.208431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.229717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.229724] Lustre: Skipped 2 previous similar messages [ 8543.285081] Lustre: Skipped 2 previous similar messages [ 8546.419412] Lustre: server umount lustre-MDT0000 complete [ 8548.324827] LustreError: 311077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.341023] LustreError: 311077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8550.622022] LustreError: 311825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848604 with bad export cookie 16178083873660534628 [ 8550.635470] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8550.647650] LustreError: 311825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8550.970752] Lustre: server umount lustre-MDT0001 complete [ 8562.517528] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8565.371859] Key type lgssc unregistered [ 8565.701276] LNet: 313647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.718235] LNetError: 313647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.737939] LNet: Removed LNI 192.168.206.122@tcp [ 8566.827547] Key type .llcrypt unregistered [ 8566.831638] Key type ._llcrypt unregistered [ 8578.234074] Key type ._llcrypt registered [ 8578.236914] Key type .llcrypt registered [ 8578.362734] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8579.508667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8579.549836] alg: No test for adler32 (adler32-zlib) [ 8580.681658] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 8580.990516] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8582.711845] Key type lgssc registered [ 8583.819929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8631.879777] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8640.577226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8640.612727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.027788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8642.075740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8642.253972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.899778] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8656.242725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.349315] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8656.381967] Lustre: 316407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8656.721749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8661.560262] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8670.973396] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.751511] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8699.090629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.404642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8699.644614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.443289] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8714.697492] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8719.327995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8719.336862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.360972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8719.844865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.845740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8719.871123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8723.417908] Lustre: server umount lustre-OST0000 complete [ 8728.544552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8728.549119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.569073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8733.987957] Lustre: server umount lustre-MDT0000 complete [ 8737.956474] LustreError: 316399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848791 with bad export cookie 11404351902827452505 [ 8737.964126] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8737.979219] LustreError: 316399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8738.380571] Lustre: server umount lustre-MDT0001 complete [ 8750.144304] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8752.777580] Key type lgssc unregistered [ 8753.053934] LNet: 319984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8753.060309] LNetError: 319984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8753.076979] LNet: Removed LNI 192.168.206.122@tcp [ 8753.943483] Key type .llcrypt unregistered [ 8753.945157] Key type ._llcrypt unregistered [ 8780.966990] Key type ._llcrypt registered [ 8780.970238] Key type .llcrypt registered [ 8781.100435] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 8782.370089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8782.402424] alg: No test for adler32 (adler32-zlib) [ 8783.416140] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 8783.614244] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 8785.343292] Key type lgssc registered [ 8786.514748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8794.514736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8794.554754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.098896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.606538] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8806.369878] LustreError: 321397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.389524] LustreError: 321397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8808.919796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.245818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8813.666932] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8820.965745] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8825.718879] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8835.196798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8835.604390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.345582] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8848.429331] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8865.249153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.255094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8865.270780] Lustre: Skipped 2 previous similar messages [ 8865.277750] Lustre: Skipped 1 previous similar message [ 8868.751598] Lustre: server umount lustre-MDT0000 complete [ 8870.378189] LustreError: 322427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.400365] LustreError: 322427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8872.583241] LustreError: 324001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848926 with bad export cookie 10092320700345200292 [ 8872.594991] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8872.604919] LustreError: 324001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8875.490145] LustreError: 321703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.494965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.506151] LustreError: 321703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8875.557825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8875.563240] Lustre: Skipped 2 previous similar messages [ 8877.537364] LustreError: 321393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8877.538074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8877.559508] LustreError: 321393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8877.586477] Lustre: Skipped 1 previous similar message [ 8879.073459] Lustre: server umount lustre-MDT0001 complete [ 8885.606668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8895.214404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.439134] LustreError: 324997:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8897.448990] LustreError: 324997:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000000d8c0b81 x1875743958147456/t0(0) o250->MGC192.168.206.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788848951 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8897.468655] LustreError: 324997:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8898.023447] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8c0f20349f65cc56 [ 8898.039973] Lustre: MGC192.168.206.122@tcp: Connection restored to 0@lo (at 0@lo) [ 8898.313770] LustreError: 323228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.414188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.481018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8902.529978] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8907.622195] LustreError: 325010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.641930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8907.649871] LustreError: 325010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8910.981768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.378840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8916.070443] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 8916.469287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8924.036871] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8925.697319] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8931.459191] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8932.813586] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8939.488336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8939.494819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.502332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.070782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8942.071837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.173216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.181183] Lustre: Skipped 2 previous similar messages [ 8952.290359] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8952.566360] Lustre: server umount lustre-OST0000 complete [ 8957.409121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8957.410287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8957.423152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.437735] Lustre: Skipped 1 previous similar message [ 8957.446585] Lustre: Skipped 3 previous similar messages [ 8962.535280] LustreError: 325012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.555192] LustreError: 325012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8962.586846] Lustre: server umount lustre-MDT0000 complete [ 8966.702235] LustreError: 321378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849020 with bad export cookie 10092320700345207894 [ 8966.711248] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8966.718406] LustreError: 321378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8967.119869] Lustre: server umount lustre-MDT0001 complete [ 8978.709944] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 8981.641236] Key type lgssc unregistered [ 8981.986282] LNet: 327684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8982.006183] LNetError: 327684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8983.080033] LNet: Removed LNI 192.168.206.122@tcp [ 8984.298774] Key type .llcrypt unregistered [ 8984.300841] Key type ._llcrypt unregistered [ 9009.260910] Key type ._llcrypt registered [ 9009.263549] Key type .llcrypt registered [ 9009.389042] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9010.350313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9010.426218] alg: No test for adler32 (adler32-zlib) [ 9011.619940] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 9011.897740] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9013.631200] Key type lgssc registered [ 9014.849344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9022.479991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9022.496918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.844356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.156953] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9033.194330] LustreError: 329097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.212272] LustreError: 329097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9037.187205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.577683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9042.011763] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9049.701421] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9055.410909] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9058.272245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9058.278643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9058.290530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.298781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.301444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.434300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9062.438426] Lustre: Skipped 2 previous similar messages [ 9064.519091] Lustre: server umount lustre-MDT0000 complete [ 9067.491086] LustreError: 329824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.500800] LustreError: 329824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9068.301106] LustreError: 330713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849122 with bad export cookie 1698715146608211604 [ 9068.319032] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.847388] Lustre: server umount lustre-MDT0001 complete [ 9084.086166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.458353] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9084.488439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9084.636807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9088.447369] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9096.711124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.923430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9096.995625] Lustre: 331682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9097.469200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9102.618320] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9109.960636] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9115.347871] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9125.740304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9125.981532] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9126.000191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9126.012904] LustreError: 333567:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9126.019537] LustreError: 333567:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9126.031223] LustreError: 333567:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9126.098792] Lustre: server umount lustre-OST0000 complete [ 9126.100866] LustreError: 333567:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9136.488040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.682672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9136.892152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.938087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9144.127895] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9152.631091] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9161.031575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9161.164205] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9161.247408] Lustre: lustre-OST0001: new disk, initializing [ 9161.250862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9161.266377] Lustre: 335109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9161.397080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9167.832530] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9170.059845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9170.067790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9170.137447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9176.145438] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9180.641660] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9180.655391] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.658993] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9180.673653] Lustre: Skipped 1 previous similar message [ 9185.582143] Lustre: server umount lustre-OST0001 complete [ 9185.764857] LustreError: 335584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.780601] LustreError: 335584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9189.856232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9189.869300] LustreError: Skipped 1 previous similar message [ 9189.872392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.889828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9189.904906] Lustre: Skipped 1 previous similar message [ 9190.884700] LustreError: 334034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9190.899371] LustreError: 334034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9195.545895] Lustre: server umount lustre-OST0000 complete [ 9200.096049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9200.123093] LustreError: Skipped 1 previous similar message [ 9200.137715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.146857] Lustre: Skipped 1 previous similar message [ 9200.152643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9200.169622] Lustre: Skipped 2 previous similar messages [ 9213.407225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9213.595606] Lustre: server umount lustre-MDT0000 complete [ 9215.458490] LustreError: 331695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.481303] LustreError: 331695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9217.116266] LustreError: 334050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849271 with bad export cookie 1698715146608212157 [ 9217.122912] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.131156] LustreError: 334050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9217.648174] Lustre: server umount lustre-MDT0001 complete [ 9226.935534] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9229.219038] Key type lgssc unregistered [ 9229.456594] LNet: 337066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9229.465234] LNetError: 337066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.489216] LNet: Removed LNI 192.168.206.122@tcp [ 9230.206189] Key type .llcrypt unregistered [ 9230.208570] Key type ._llcrypt unregistered [ 9290.745188] Key type ._llcrypt registered [ 9290.748932] Key type .llcrypt registered [ 9290.847162] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [ 9303.221791] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9304.080244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9304.096601] alg: No test for adler32 (adler32-zlib) [ 9305.133059] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 9305.350938] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9307.007209] Key type lgssc registered [ 9307.892374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9335.256486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9335.276527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.445439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9336.465203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9336.547538] Lustre: lustre-MDT0000: new disk, initializing [ 9336.602108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.622955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9339.828407] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9348.254515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.389886] Lustre: 341975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9348.435691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9348.442198] Lustre: Skipped 1 previous similar message [ 9348.516752] Lustre: lustre-MDT0001: new disk, initializing [ 9348.617981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9348.662343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9348.670412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9352.091020] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9360.575826] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9365.234450] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9371.041272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9371.270977] Lustre: lustre-OST0000: new disk, initializing [ 9371.279951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9371.287110] Lustre: 343943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9371.385854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9376.994839] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9378.850840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9378.859982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9378.963115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9384.754036] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9388.185672] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9388.377088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9392.083124] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9392.232269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9394.144244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.151630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.158703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.159839] Lustre: Skipped 1 previous similar message [ 9399.271843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.279390] Lustre: Skipped 2 previous similar messages [ 9399.804465] Lustre: server umount lustre-OST0000 complete [ 9404.385090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.399532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.402261] Lustre: Skipped 1 previous similar message [ 9409.508825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.517964] Lustre: Skipped 1 previous similar message [ 9414.625450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9414.634619] Lustre: Skipped 1 previous similar message [ 9417.183140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9417.358751] Lustre: server umount lustre-MDT0000 complete [ 9420.265797] LustreError: 342778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849474 with bad export cookie 2791535785163309876 [ 9420.267494] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9420.272935] LustreError: 342778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9420.477616] Lustre: server umount lustre-MDT0001 complete [ 9426.597353] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:37:59 (1788849479) [ 9433.194427] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:38:05 (1788849485) [ 9440.355856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.747734] LustreError: 346698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.773759] LustreError: 346698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9440.820321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9444.237557] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9445.857878] LustreError: 346699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.772043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.981629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9451.090741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9454.901189] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9460.964893] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9465.611517] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9471.522405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9471.805051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.727337] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9483.804647] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9494.917863] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9496.190889] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9497.673026] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9499.002613] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9500.414130] Lustre: Failing over lustre-MDT0000 [ 9500.642336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.650163] LustreError: 348538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.651847] Lustre: Skipped 2 previous similar messages [ 9500.672626] Lustre: server umount lustre-MDT0000 complete [ 9505.116463] LustreError: 346695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9505.129357] LustreError: 346695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9510.196139] LustreError: 346694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.216056] LustreError: 346694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9517.686471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.795591] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9518.008322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9518.058398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9520.436262] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9522.287607] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9523.187551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9523.217449] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9523.262446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9530.878781] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9532.223974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9535.242576] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9536.942275] Lustre: Failing over lustre-MDT0000 [ 9537.175496] Lustre: server umount lustre-MDT0000 complete [ 9538.531555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.536951] LustreError: 346693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9538.542041] Lustre: Skipped 3 previous similar messages [ 9538.557190] LustreError: 346693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9554.316044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.410383] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9554.587275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.625800] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9556.288362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9558.480328] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9560.053045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9560.064394] Lustre: Skipped 2 previous similar messages [ 9560.104570] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9560.146429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9565.929819] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9567.110609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9575.398539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.402311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9575.419236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9575.424634] Lustre: Skipped 1 previous similar message [ 9578.267604] Lustre: server umount lustre-MDT0000 complete [ 9580.513430] LustreError: 346693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.522800] LustreError: 346693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9580.840169] LustreError: 350680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849634 with bad export cookie 2791535785163312585 [ 9580.841499] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9580.850918] LustreError: 350680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9581.081672] Lustre: server umount lustre-MDT0001 complete [ 9594.673811] Lustre: server umount lustre-OST0000 complete [ 9607.770739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.698213] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9613.282614] LustreError: 353052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.295093] LustreError: 353052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9618.306556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.598689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9618.602580] Lustre: Skipped 1 previous similar message [ 9621.905481] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9627.410877] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.109334] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.138277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9641.630567] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9647.063509] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9653.741286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9663.969731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9663.970781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.973165] LustreError: Skipped 1 previous similar message [ 9663.979145] Lustre: Skipped 1 previous similar message [ 9663.980553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9663.992267] Lustre: Skipped 3 previous similar messages [ 9667.037794] Lustre: server umount lustre-OST0000 complete [ 9670.112136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9676.240803] Lustre: server umount lustre-MDT0000 complete [ 9678.823099] LustreError: 353032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849732 with bad export cookie 2791535785163313334 [ 9678.826462] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9678.832583] LustreError: 353032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9679.078185] Lustre: server umount lustre-MDT0001 complete [ 9686.626944] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9688.745891] Key type lgssc unregistered [ 9688.984206] LNet: 356603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9688.991268] LNetError: 356603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9689.003327] LNet: Removed LNI 192.168.206.122@tcp [ 9689.643583] Key type .llcrypt unregistered [ 9689.645860] Key type ._llcrypt unregistered [ 9708.981805] Key type ._llcrypt registered [ 9708.983487] Key type .llcrypt registered [ 9709.053358] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9709.838799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9709.905485] alg: No test for adler32 (adler32-zlib) [ 9710.913707] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 9711.222767] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9712.879175] Key type lgssc registered [ 9713.735131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9719.442988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9719.463963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.773651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.959876] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9730.582029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.816497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9734.279970] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9739.885187] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9743.853886] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9748.471664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.679147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9753.231412] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9758.967172] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9766.125185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9777.120571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9777.127641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.137929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.729852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.730385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.747498] Lustre: Skipped 2 previous similar messages [ 9783.266251] Lustre: server umount lustre-MDT0000 complete [ 9785.715562] LustreError: 358076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849839 with bad export cookie 14914460108300522283 [ 9785.716711] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9785.725830] LustreError: 358076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9785.921909] Lustre: server umount lustre-MDT0001 complete [ 9798.908785] Lustre: server umount lustre-OST0000 complete [ 9808.728619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.013487] LustreError: 361822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.028744] LustreError: 361822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9809.062914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.886901] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9814.497462] LustreError: 361823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.350944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.594241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9818.706985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9822.078101] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9828.336974] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9832.610777] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9837.088195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.311920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9841.705444] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9847.311416] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9853.740943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9869.285358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9869.286940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9869.308937] Lustre: Skipped 3 previous similar messages [ 9869.318503] Lustre: Skipped 1 previous similar message [ 9873.109818] Lustre: server umount lustre-MDT0000 complete [ 9874.400806] LustreError: 361818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.412294] LustreError: 361818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9875.522192] LustreError: 361802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849929 with bad export cookie 14914460108300523396 [ 9875.524944] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9875.530260] LustreError: 361802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9875.752550] Lustre: server umount lustre-MDT0001 complete [ 9888.711701] Lustre: server umount lustre-OST0000 complete [ 9898.393447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.679339] LustreError: 365752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.733821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.709131] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9904.098284] LustreError: 365753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.120712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.107990] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9915.182272] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9918.789644] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9922.803770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9924.072872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9926.905704] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [ 9931.956752] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9937.381655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.382684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.386836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9937.386842] Lustre: Skipped 1 previous similar message [ 9937.396911] Lustre: Skipped 1 previous similar message [ 9940.792538] Lustre: server umount lustre-OST0000 complete [ 9943.520436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9943.526775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.621285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.628044] Lustre: Skipped 3 previous similar messages [ 9949.342664] Lustre: server umount lustre-MDT0000 complete [ 9951.643395] LustreError: 365735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850005 with bad export cookie 14914460108300524418 [ 9951.652293] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9951.652855] LustreError: 365735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9951.899989] Lustre: server umount lustre-MDT0001 complete [ 9958.960056] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [ 9960.648661] Key type lgssc unregistered [ 9960.870434] LNet: 369279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9960.876220] LNetError: 369279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9960.886664] LNet: Removed LNI 192.168.206.122@tcp [ 9961.473160] Key type .llcrypt unregistered [ 9961.475778] Key type ._llcrypt unregistered [ 9977.931837] Key type ._llcrypt registered [ 9977.934683] Key type .llcrypt registered [ 9977.998848] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [ 9978.548487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9978.684161] alg: No test for adler32 (adler32-zlib) [ 9979.613237] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 9979.765635] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [ 9981.391412] Key type lgssc registered [ 9982.155313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9987.787494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9987.803025] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.808972] LustreError: 370696:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9987.827638] LustreError: 370696:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9987.832794] LustreError: 370696:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9987.837466] LustreError: 370696:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9987.843083] LustreError: 370696:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9993.376547] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.386037] LustreError: 371093:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9993.404585] LustreError: 371093:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9993.409819] LustreError: 371093:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9993.414809] LustreError: 371093:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9993.419401] LustreError: 371093:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10006.627447] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10008.581860] Key type lgssc unregistered [10008.799952] LNet: 372094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10008.807704] LNetError: 372094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10008.828062] LNet: Removed LNI 192.168.206.122@tcp [10009.409405] Key type .llcrypt unregistered [10009.411165] Key type ._llcrypt unregistered [10021.728347] Key type ._llcrypt registered [10021.731565] Key type .llcrypt registered [10021.818538] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [10032.372742] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10033.028629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10033.099062] alg: No test for adler32 (adler32-zlib) [10034.087896] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10034.277767] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10035.903163] Key type lgssc registered [10036.754045] Lustre: Echo OBD driver; http://www.lustre.org/ [10059.777272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10059.789658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.941592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10060.963405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10061.017959] Lustre: lustre-MDT0000: new disk, initializing [10061.062252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.076938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10063.935409] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10071.316070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.403297] Lustre: 375223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10071.448701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10071.451799] Lustre: Skipped 1 previous similar message [10071.508384] Lustre: lustre-MDT0001: new disk, initializing [10071.591132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10071.627789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10071.646985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10075.220939] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10081.791417] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10085.236368] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10089.634208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10089.815484] Lustre: lustre-OST0000: new disk, initializing [10089.818850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10089.822960] Lustre: 377188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10089.888336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10093.936076] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10099.759908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10099.774490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10099.832037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10100.716583] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10103.832502] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10103.987240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10106.723757] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10106.853175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10109.920710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10109.933898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.948909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.439185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.442083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10110.457779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.469919] Lustre: Skipped 1 previous similar message [10115.558411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10115.564828] Lustre: Skipped 1 previous similar message [10120.675900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10122.207128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10122.279424] Lustre: server umount lustre-OST0000 complete [10125.793659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.794874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.802882] Lustre: Skipped 1 previous similar message [10125.808494] Lustre: Skipped 2 previous similar messages [10130.494257] Lustre: server umount lustre-MDT0000 complete [10132.301573] LustreError: 375215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850186 with bad export cookie 7277774386631596470 [10132.305970] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10132.307427] LustreError: 375215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10132.469916] Lustre: server umount lustre-MDT0001 complete [10136.309316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10137.113430] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:49:50 (1788850190) [10140.765344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.026940] LustreError: 379646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.039558] LustreError: 379646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10141.077558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10143.722542] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10146.276035] LustreError: 379647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.145830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.386890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10152.290448] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10157.322674] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10160.912566] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10165.112167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.345107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10169.395390] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10173.874300] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10177.926332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10178.028744] Lustre: lustre-OST0001: new disk, initializing [10178.032147] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10178.036819] Lustre: 382482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10181.553464] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10187.307093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10187.316251] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10187.382161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10187.553358] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10192.359377] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.360472] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10192.364157] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10192.373425] Lustre: Skipped 1 previous similar message [10196.189922] Lustre: server umount lustre-OST0001 complete [10197.478941] LustreError: 383289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.487117] LustreError: 383289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10202.597961] LustreError: 381465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.609560] LustreError: 381465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10207.720118] LustreError: 382167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.729299] LustreError: 382167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10210.785260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10210.790758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.799992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10210.803991] Lustre: Skipped 2 previous similar messages [10216.055702] Lustre: server umount lustre-OST0000 complete [10221.024210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10221.029330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.035606] Lustre: Skipped 1 previous similar message [10224.581276] Lustre: server umount lustre-MDT0000 complete [10226.603577] LustreError: 379627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850280 with bad export cookie 7277774386631597352 [10226.607546] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10226.609121] LustreError: 379627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10232.913990] Lustre: server umount lustre-MDT0001 complete [10238.841240] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10240.429583] Key type lgssc unregistered [10240.630450] LNet: 384463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10240.633940] LNetError: 384463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10240.644600] LNet: Removed LNI 192.168.206.122@tcp [10241.095548] Key type .llcrypt unregistered [10241.097179] Key type ._llcrypt unregistered [10270.862235] Key type ._llcrypt registered [10270.865662] Key type .llcrypt registered [10270.928983] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10271.665552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10271.693055] alg: No test for adler32 (adler32-zlib) [10272.622491] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10272.773481] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10274.391155] Key type lgssc registered [10275.157260] Lustre: Echo OBD driver; http://www.lustre.org/ [10280.114872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10280.125505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10281.287477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10281.309352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10281.415647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10283.951177] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10289.015835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.079988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10289.110730] Lustre: 387236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10289.329583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10292.075608] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10296.571716] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10299.928908] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10303.972415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10303.978996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10303.986573] Lustre: Skipped 1 previous similar message [10303.992994] Lustre: Skipped 1 previous similar message [10307.438931] Lustre: server umount lustre-MDT0000 complete [10309.967102] LustreError: 387227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850363 with bad export cookie 7862342673388743553 [10309.972298] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10309.980526] LustreError: 387227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10310.198850] Lustre: server umount lustre-MDT0001 complete [10315.900432] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:52:48 (1788850368) [10319.877294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10320.223973] LustreError: 389746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.238690] LustreError: 389746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10320.274193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.993122] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10325.475044] LustreError: 389747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.478283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.727284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10331.878249] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10337.096596] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10340.641511] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10345.399132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.526951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10345.657915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10349.781630] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10354.559955] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10360.556656] Lustre: Permanently deactivating lustre-OST0000 [10360.566806] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10367.994048] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10369.107115] LustreError: 392779:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10369.123672] LustreError: 392779:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10369.130365] LustreError: 392779:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10370.392772] LustreError: 392827:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10370.399053] LustreError: 392827:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10377.656821] Lustre: server umount lustre-OST0000 complete [10379.134512] LustreError: 393077:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10379.141942] LustreError: 393077:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10380.256050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10380.262661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.269737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.865893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10384.866591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.876852] Lustre: Skipped 1 previous similar message [10389.989773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10389.993241] Lustre: Skipped 1 previous similar message [10391.051692] Lustre: server umount lustre-MDT0000 complete [10393.071459] LustreError: 389726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850447 with bad export cookie 7862342673388744134 [10393.076538] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.079774] LustreError: 389726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10403.380427] Lustre: server umount lustre-MDT0001 complete [10408.088141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10410.467398] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10411.879139] LustreError: 394233:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10415.021423] Lustre: 394572:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.122@tcp [10416.119770] Lustre: server umount MGS complete [10422.524345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.766332] LustreError: 395296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.781363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10422.786438] Lustre: Skipped 1 previous similar message [10422.816135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10425.278965] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10427.872757] LustreError: 395297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.326065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.525160] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10433.161301] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10437.256894] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.246244] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10441.239030] Lustre: Permanently reactivating lustre-OST0000 [10441.257612] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10441.261671] Lustre: Skipped 2 previous similar messages [10468.832640] LustreError: 397120:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10473.594344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.842788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10473.849758] Lustre: Skipped 1 previous similar message [10477.054451] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10480.979530] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10489.315649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.318421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10489.326820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10489.327992] Lustre: Skipped 1 previous similar message [10489.340686] LustreError: Skipped 1 previous similar message [10491.114165] Lustre: server umount lustre-OST0000 complete [10493.342210] Lustre: server umount lustre-MDT0000 complete [10494.433051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.434372] LustreError: 395291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.440015] Lustre: Skipped 1 previous similar message [10494.459112] LustreError: 395291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10495.266236] LustreError: 395277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850549 with bad export cookie 7862342673388745170 [10495.271858] LustreError: 395277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10495.272026] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.438747] Lustre: server umount lustre-MDT0001 complete [10500.733391] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10502.273188] Key type lgssc unregistered [10502.470494] LNet: 399894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10502.474774] LNetError: 399894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10503.525426] LNet: Removed LNI 192.168.206.122@tcp [10503.963515] Key type .llcrypt unregistered [10503.965297] Key type ._llcrypt unregistered [10512.739230] Key type ._llcrypt registered [10512.741069] Key type .llcrypt registered [10512.796650] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [10520.559962] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10520.906033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10520.925280] alg: No test for adler32 (adler32-zlib) [10521.826790] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10521.939840] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10523.535354] Key type lgssc registered [10524.132483] Lustre: Echo OBD driver; http://www.lustre.org/ [10540.619878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10540.628288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.741600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10541.759357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10541.812374] Lustre: lustre-MDT0000: new disk, initializing [10541.851025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10541.858892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10543.907405] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10549.444875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.503288] Lustre: 403020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10549.535240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10549.538693] Lustre: Skipped 1 previous similar message [10549.582865] Lustre: lustre-MDT0001: new disk, initializing [10549.625159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10549.644818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10549.649869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10551.529800] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10556.065903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10558.345783] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10561.491109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.633483] Lustre: lustre-OST0000: new disk, initializing [10561.636551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10561.641362] Lustre: 404989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10561.678924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10564.268240] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10566.673254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10566.681927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10566.722124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10568.973306] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10570.740450] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10570.834667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10572.577287] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10572.675303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10576.867799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.868273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10576.868593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10576.874500] Lustre: Skipped 1 previous similar message [10579.614722] Lustre: server umount lustre-OST0000 complete [10581.984806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.985536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.992028] Lustre: Skipped 1 previous similar message [10581.995796] Lustre: Skipped 1 previous similar message [10587.105450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.108151] Lustre: Skipped 1 previous similar message [10587.434447] Lustre: server umount lustre-MDT0000 complete [10588.958320] LustreError: 403816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850642 with bad export cookie 9052655833893166101 [10588.960296] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.963193] LustreError: 403816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10589.100724] Lustre: server umount lustre-MDT0001 complete [10592.488961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10593.154970] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:57:26 (1788850646) [10596.213312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.483718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10598.261305] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10601.796297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.955242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.666137] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10606.658578] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10608.980172] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10611.849959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10611.998445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10614.522428] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10617.861477] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10618.753608] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10625.946832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10636.769736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10636.776321] LustreError: Skipped 1 previous similar message [10636.776555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.781650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10636.787994] Lustre: Skipped 1 previous similar message [10638.890739] Lustre: server umount lustre-OST0000 complete [10641.888797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.891387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10641.893845] Lustre: Skipped 1 previous similar message [10641.899989] Lustre: Skipped 2 previous similar messages [10652.128910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10652.132437] Lustre: Skipped 3 previous similar messages [10655.199156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10655.312105] Lustre: server umount lustre-MDT0000 complete [10656.225306] LustreError: 408174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10656.236632] LustreError: 408174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10657.311183] LustreError: 407426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850711 with bad export cookie 9052655833893166983 [10657.316146] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10657.317578] LustreError: 407426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10657.503312] Lustre: server umount lustre-MDT0001 complete [10663.485503] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10664.873490] Key type lgssc unregistered [10665.019731] LNet: 411191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10665.024956] LNetError: 411191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10665.033597] LNet: Removed LNI 192.168.206.122@tcp [10665.471314] Key type .llcrypt unregistered [10665.473191] Key type ._llcrypt unregistered [10675.862867] Key type ._llcrypt registered [10675.864425] Key type .llcrypt registered [10675.915963] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10676.036321] Key type .llcrypt unregistered [10676.038768] Key type ._llcrypt unregistered [10684.203978] Key type ._llcrypt registered [10684.205743] Key type .llcrypt registered [10684.250636] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10684.702856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10684.719105] alg: No test for adler32 (adler32-zlib) [10685.608933] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10685.734349] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10687.327323] Key type lgssc registered [10687.788764] Lustre: Echo OBD driver; http://www.lustre.org/ [10690.856522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10690.867491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.047604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10693.684951] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10697.216283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10697.356486] LustreError: 413367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.365787] LustreError: 413367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10697.367268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10699.779580] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10701.472399] LustreError: 413371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10701.478849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10701.481821] LustreError: 413371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10702.950835] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10705.950779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.151192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10707.958271] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10716.642680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10716.645796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10716.649117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10721.761990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10721.765212] Lustre: Skipped 1 previous similar message [10722.156328] Lustre: server umount lustre-OST0000 complete [10725.861759] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10725.868520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.878914] Lustre: Skipped 1 previous similar message [10725.882118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10725.886147] Lustre: Skipped 1 previous similar message [10726.882290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10730.091596] Lustre: server umount lustre-MDT0000 complete [10732.000692] LustreError: 415116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.011254] LustreError: 415116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10732.054301] LustreError: 413351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850786 with bad export cookie 9594743924522914882 [10732.057509] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10732.060158] LustreError: 413351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10732.206364] Lustre: server umount lustre-MDT0001 complete [10736.792316] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10738.080753] Key type lgssc unregistered [10738.240424] LNet: 416513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10738.246236] LNetError: 416513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10738.256416] LNet: Removed LNI 192.168.206.122@tcp [10738.656353] Key type .llcrypt unregistered [10738.657980] Key type ._llcrypt unregistered [10749.909459] Key type ._llcrypt registered [10749.910772] Key type .llcrypt registered [10749.955777] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10750.325464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10750.341887] alg: No test for adler32 (adler32-zlib) [10751.212153] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10751.325530] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10752.927180] Key type lgssc registered [10753.395442] Lustre: Echo OBD driver; http://www.lustre.org/ [10756.819681] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.828853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10772.319519] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10777.439465] LustreError: 417892:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.122@tcp: failed processing log, type 4: rc = -110 [10804.064440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10807.724943] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10811.722207] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10815.425878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.738690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10815.786735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10818.149622] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10822.618672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.842277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10824.945795] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10828.853190] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10831.566333] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10836.447972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10836.451375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10836.456888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10838.497692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10838.501666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10838.504349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.493512] Lustre: server umount lustre-OST0000 complete [10843.617773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.618713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10843.619062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10843.630591] Lustre: Skipped 1 previous similar message [10843.634430] Lustre: Skipped 1 previous similar message [10848.372732] Lustre: server umount lustre-MDT0000 complete [10848.742974] LustreError: 418942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10848.754378] LustreError: 418942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10850.889801] LustreError: 417898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850904 with bad export cookie 12607986620632268258 [10850.893380] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10850.894983] LustreError: 417898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10851.104269] Lustre: server umount lustre-MDT0001 complete [10857.225644] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10858.693253] Key type lgssc unregistered [10858.860801] LNet: 421473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10858.864753] LNetError: 421473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10858.891827] LNet: Removed LNI 192.168.206.122@tcp [10859.376469] Key type .llcrypt unregistered [10859.378231] Key type ._llcrypt unregistered [10872.200946] Key type ._llcrypt registered [10872.202671] Key type .llcrypt registered [10872.249206] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10872.744534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10872.770414] alg: No test for adler32 (adler32-zlib) [10873.656936] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10873.756528] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10875.343866] Key type lgssc registered [10875.857110] Lustre: Echo OBD driver; http://www.lustre.org/ [10879.599800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10879.611082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.820213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.679876] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10886.734023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10886.913800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10888.595443] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10891.463024] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10893.766176] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10896.785374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10896.951926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10899.407096] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10902.641653] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10905.127940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10910.178452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.181372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10910.183408] Lustre: Skipped 2 previous similar messages [10910.190919] Lustre: Skipped 2 previous similar messages [10911.201565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10911.208913] Lustre: Skipped 2 previous similar messages [10914.112288] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.22@tcp (stopping) [10916.080953] Lustre: server umount lustre-MDT0000 complete [10916.321416] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.331701] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10917.239297] LustreError: 422844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.22@tcp arrived at 1788850971 with bad export cookie 10398022625334087629 [10917.251601] LustreError: 422844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10918.194101] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850972 with bad export cookie 10398022625334086985 [10918.199096] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.199149] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10921.441912] LustreError: 423603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.443872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10921.454886] LustreError: 423603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10921.466846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10923.488682] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.495326] LustreError: 422865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10924.325222] Lustre: server umount lustre-OST0000 complete [10927.537177] LustreError: 423591:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000147d496f x1875746149655296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10927.618421] Lustre: server umount lustre-MDT0001 complete [10932.353392] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [10933.647214] Key type lgssc unregistered [10933.805488] LNet: 426544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10933.808334] LNetError: 426544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10933.819944] LNet: Removed LNI 192.168.206.122@tcp [10934.262243] Key type .llcrypt unregistered [10934.263580] Key type ._llcrypt unregistered [10946.389755] Key type ._llcrypt registered [10946.391209] Key type .llcrypt registered [10946.455181] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [10946.937098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10946.984874] alg: No test for adler32 (adler32-zlib) [10947.901170] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [10948.039548] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [10949.631184] Key type lgssc registered [10950.165549] Lustre: Echo OBD driver; http://www.lustre.org/ [10954.171701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10954.187714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.438412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.471544] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10961.313682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.508361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10963.245114] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10966.659653] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10969.074404] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10972.441088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.641397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10975.648139] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [10979.204773] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10980.839643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10986.684169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10986.691056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10990.897389] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.22@tcp (stopping) [10991.583905] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10991.587251] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.097354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10992.100088] Lustre: Skipped 2 previous similar messages [10992.863880] Lustre: server umount lustre-MDT0001 complete [10996.193221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10996.197582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.204065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10997.220250] LustreError: 427933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.229551] LustreError: 427933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11001.184733] Lustre: server umount lustre-OST0000 complete [11002.337232] LustreError: 427934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11002.733688] Lustre: server umount lustre-MDT0000 complete [11007.882561] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11009.045461] Key type lgssc unregistered [11009.181647] LNet: 431616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11009.184888] LNetError: 431616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11009.192351] LNet: Removed LNI 192.168.206.122@tcp [11009.566879] Key type .llcrypt unregistered [11009.568473] Key type ._llcrypt unregistered [11018.560766] Key type ._llcrypt registered [11018.562093] Key type .llcrypt registered [11018.609515] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [11025.341346] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11025.852602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11025.889176] alg: No test for adler32 (adler32-zlib) [11026.756252] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [11026.865889] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11028.456085] Key type lgssc registered [11028.908024] Lustre: Echo OBD driver; http://www.lustre.org/ [11043.281556] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11049.160467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11049.167763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.260910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11050.273455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11050.313176] Lustre: lustre-MDT0000: new disk, initializing [11050.337193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11050.345788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11051.821856] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11058.428625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11058.556118] Lustre: lustre-OST0000: new disk, initializing [11058.559169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11058.562707] Lustre: Skipped 1 previous similar message [11058.566123] Lustre: 436616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11058.600581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11060.843579] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11063.819344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11063.823431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11063.834349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11064.908189] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11070.984956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.035248] Lustre: 435596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11071.062527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11071.103403] Lustre: lustre-MDT0001: new disk, initializing [11071.148908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11071.166131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11071.170454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11072.710784] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11076.334539] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11076.410548] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11077.776514] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11077.850202] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11081.619632] Lustre: server umount lustre-MDT0001 complete [11082.207791] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11082.210624] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.216712] Lustre: Skipped 1 previous similar message [11087.329155] LustreError: 435604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.336361] LustreError: 435604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11092.449116] LustreError: 435608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.454210] LustreError: 435608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11094.495944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11094.499345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.503930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.568592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.569723] LustreError: 435609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.688977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.689885] LustreError: 435608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11107.809210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11107.809841] LustreError: 435609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.319122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11108.379478] Lustre: server umount lustre-OST0000 complete [11124.191138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11124.291803] Lustre: server umount lustre-MDT0000 complete [11129.126156] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11130.197173] Key type lgssc unregistered [11130.330496] LNet: 440361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11130.333521] LNetError: 440361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11130.342433] LNet: Removed LNI 192.168.206.122@tcp [11130.674863] Key type .llcrypt unregistered [11130.676437] Key type ._llcrypt unregistered [11143.805737] Key type ._llcrypt registered [11143.807023] Key type .llcrypt registered [11143.854516] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [11143.961548] Key type .llcrypt unregistered [11143.963222] Key type ._llcrypt unregistered [11150.840074] Key type ._llcrypt registered [11150.841574] Key type .llcrypt registered [11150.885518] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11151.251774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11151.286791] alg: No test for adler32 (adler32-zlib) [11152.147058] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [11152.244934] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [11153.831218] Key type lgssc registered [11154.221922] Lustre: Echo OBD driver; http://www.lustre.org/ [11163.273652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11163.280261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.374800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11164.383662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11164.421936] Lustre: lustre-MDT0000: new disk, initializing [11164.445614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.452637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11165.797474] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11170.057362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.101265] Lustre: 443700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11170.130606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11170.133200] Lustre: Skipped 1 previous similar message [11170.172156] Lustre: lustre-MDT0001: new disk, initializing [11170.215983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11170.230727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11170.235624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11171.626823] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11175.611579] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11178.071438] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11181.089290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.211426] Lustre: lustre-OST0000: new disk, initializing [11181.213435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11181.217199] Lustre: 445669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11181.253768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11183.552611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11183.557810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11183.606678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11183.803908] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11188.165200] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11193.826571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.828207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11193.831462] Lustre: Skipped 2 previous similar messages [11193.836575] Lustre: Skipped 2 previous similar messages [11196.956488] Lustre: server umount lustre-MDT0000 complete [11198.625972] LustreError: 444495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851252 with bad export cookie 15992170795158988835 [11198.627256] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11198.630529] LustreError: 444495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11198.746405] Lustre: server umount lustre-MDT0001 complete [11210.415716] Lustre: server umount lustre-OST0000 complete [11215.373633] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:07:48 (1788851268) [11219.025394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.273228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.951360] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11224.544261] LustreError: 447694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.552932] LustreError: 447694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11224.560322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.737987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11226.640574] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11229.539592] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11231.629992] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11247.394205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.476072] Lustre: Found index 0 for lustre-OST0000, updating log [11247.541961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11249.875485] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11252.509665] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11265.958751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11270.112528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11270.115675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.120853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.140303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.142744] Lustre: Skipped 1 previous similar message [11275.868859] Lustre: server umount lustre-OST0000 complete [11288.893659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.894702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.899168] Lustre: Skipped 2 previous similar messages [11288.904507] Lustre: Skipped 2 previous similar messages [11294.176482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11294.179405] Lustre: Skipped 1 previous similar message [11303.391630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11303.392178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11303.398499] Lustre: Skipped 5 previous similar messages [11303.469136] Lustre: server umount lustre-MDT0000 complete [11305.167113] LustreError: 447676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851359 with bad export cookie 15992170795158990228 [11305.168624] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11305.173563] LustreError: 447676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11305.319644] Lustre: server umount lustre-MDT0001 complete [11308.817786] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:09:21 (1788851361) [11313.055700] LNet: 1 local NIs in recovery (showing 1): 192.168.206.122@tcp [11895.484331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11895.762872] LustreError: 454117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.801076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11897.798403] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11900.898357] LustreError: 454118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.257596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.448934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11904.482809] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11908.129418] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11910.304903] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11925.932489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11926.065593] Lustre: Found index 0 for lustre-OST0000, updating log [11926.231351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11930.654397] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [11934.892109] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11939.812728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.814045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11939.821282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11942.137105] Lustre: server umount lustre-OST0000 complete [11945.442371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.450900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.451211] Lustre: Skipped 1 previous similar message [11949.537718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11949.541683] Lustre: Skipped 8 previous similar messages [11950.612413] Lustre: server umount lustre-MDT0000 complete [11951.586512] LustreError: 454118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.605747] LustreError: 454118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11952.831231] LustreError: 455976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852006 with bad export cookie 15992170795158991299 [11952.841976] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11953.017648] Lustre: server umount lustre-MDT0001 complete [11953.060404] LNet: 1 local NIs in recovery (showing 1): 192.168.206.122@tcp [11971.937695] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [11976.061211] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [11991.905288] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:20:44 (1788852044) [11999.236463] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12000.455052] Key type lgssc unregistered [12000.597761] LNet: 460724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12000.601613] LNetError: 460724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12055.913081] LNet: Removed LNI 192.168.206.122@tcp [12056.330983] Key type .llcrypt unregistered [12056.332618] Key type ._llcrypt unregistered [12063.593408] Key type ._llcrypt registered [12063.594516] Key type .llcrypt registered [12063.634143] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12063.970554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12063.985063] alg: No test for adler32 (adler32-zlib) [12064.853164] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12064.961027] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12066.543179] Key type lgssc registered [12066.950665] Lustre: Echo OBD driver; http://www.lustre.org/ [12068.559941] LNet: Added LNI 192.168.206.122@tcp100 [8/256/0/180] [12068.912196] LNet: Added LNI 192.168.206.122@tcp101 [8/256/0/180] [12069.269913] LNet: Added LNI 192.168.206.122@tcp102 [8/256/0/180] [12069.608109] LNet: Added LNI 192.168.206.122@tcp103 [8/256/0/180] [12069.977868] LNet: Added LNI 192.168.206.122@tcp104 [8/256/0/180] [12070.316024] LNet: Added LNI 192.168.206.122@tcp105 [8/256/0/180] [12070.664645] LNet: Added LNI 192.168.206.122@tcp106 [8/256/0/180] [12071.000742] LNet: Added LNI 192.168.206.122@tcp107 [8/256/0/180] [12071.336824] LNet: Added LNI 192.168.206.122@tcp108 [8/256/0/180] [12071.675703] LNet: Added LNI 192.168.206.122@tcp109 [8/256/0/180] [12072.072575] LNet: Added LNI 192.168.206.122@tcp110 [8/256/0/180] [12072.432611] LNet: Added LNI 192.168.206.122@tcp111 [8/256/0/180] [12072.799142] LNet: Added LNI 192.168.206.122@tcp112 [8/256/0/180] [12073.146894] LNet: Added LNI 192.168.206.122@tcp113 [8/256/0/180] [12073.498840] LNet: Added LNI 192.168.206.122@tcp114 [8/256/0/180] [12073.868345] LNet: Added LNI 192.168.206.122@tcp115 [8/256/0/180] [12074.214597] LNet: Added LNI 192.168.206.122@tcp116 [8/256/0/180] [12074.591737] LNet: Added LNI 192.168.206.122@tcp117 [8/256/0/180] [12074.948936] LNet: Added LNI 192.168.206.122@tcp118 [8/256/0/180] [12075.332773] LNet: Added LNI 192.168.206.122@tcp119 [8/256/0/180] [12075.753350] LNet: Added LNI 192.168.206.122@tcp120 [8/256/0/180] [12076.182798] LNet: Added LNI 192.168.206.122@tcp121 [8/256/0/180] [12076.572673] LNet: Added LNI 192.168.206.122@tcp122 [8/256/0/180] [12076.990807] LNet: Added LNI 192.168.206.122@tcp123 [8/256/0/180] [12077.346290] LNet: Added LNI 192.168.206.122@tcp124 [8/256/0/180] [12077.706040] LNet: Added LNI 192.168.206.122@tcp125 [8/256/0/180] [12078.049695] LNet: Added LNI 192.168.206.122@tcp126 [8/256/0/180] [12078.404020] LNet: Added LNI 192.168.206.122@tcp127 [8/256/0/180] [12078.767935] LNet: Added LNI 192.168.206.122@tcp128 [8/256/0/180] [12079.123488] LNet: Added LNI 192.168.206.122@tcp129 [8/256/0/180] [12079.501233] LNet: Added LNI 192.168.206.122@tcp130 [8/256/0/180] [12079.927174] LNet: Added LNI 192.168.206.122@tcp131 [8/256/0/180] [12080.407351] LNet: Added LNI 192.168.206.122@tcp132 [8/256/0/180] [12080.801609] LNet: Added LNI 192.168.206.122@tcp133 [8/256/0/180] [12081.169973] LNet: Added LNI 192.168.206.122@tcp134 [8/256/0/180] [12081.507100] LNet: Added LNI 192.168.206.122@tcp135 [8/256/0/180] [12084.500186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12084.508624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.564195] Lustre: 463716:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [12085.588232] LNetError: 463737:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12085.598500] LNetError: Protocol error connecting to 192.168.206.122@tcp112 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12085.655157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12085.675798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12085.716400] Lustre: lustre-MDT0000: new disk, initializing [12085.773172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12085.783234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12086.624035] LNetError: 461181:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12086.631213] LNetError: Protocol error connecting to 192.168.206.122@tcp101 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12088.100639] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12094.647509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.734754] Lustre: 463733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12094.780031] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12094.783237] Lustre: Skipped 1 previous similar message [12094.832496] Lustre: lustre-MDT0001: new disk, initializing [12094.857861] Lustre: 464535:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [12094.864574] Lustre: 464535:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12094.957922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12094.987583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12094.994081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12097.835522] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12104.008845] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12107.802700] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12111.317773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12111.473872] Lustre: lustre-OST0000: new disk, initializing [12111.476395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12111.480906] Lustre: 465704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12111.533766] Lustre: 465709:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [12111.536989] Lustre: 465709:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [12111.563826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12114.526850] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12116.990689] Lustre: 466217:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.122@tcp: can't add NID 192.168.206.122@tcp130: rc = -75 [12116.995730] Lustre: 466217:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12117.037537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12117.045170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12117.119606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12117.219385] LNetError: 463736:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12117.225963] LNetError: Protocol error connecting to 192.168.206.122@tcp101 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12119.394196] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12120.288361] LNetError: 461180:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12120.293961] LNetError: Protocol error connecting to 192.168.206.122@tcp112 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12122.595177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12122.599164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.606047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12127.200183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12127.205783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.212452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12128.277592] Lustre: server umount lustre-OST0000 complete [12130.528550] LNetError: 463738:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12130.534395] LNetError: Protocol error connecting to 192.168.206.122@tcp101 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12130.783738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12130.787145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.792187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12130.795634] Lustre: Skipped 1 previous similar message [12132.831928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12132.837713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.877671] Lustre: server umount lustre-MDT0000 complete [12137.675455] LustreError: 463724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852191 with bad export cookie 18122607325497218203 [12137.676974] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12137.680685] LustreError: 463724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12137.847841] Lustre: server umount lustre-MDT0001 complete [12142.824118] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12143.845770] LNetError: 463739:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.122 [12143.852236] LNetError: 463739:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [12143.858448] LNetError: Protocol error connecting to 192.168.206.122@tcp101 at host 192.168.206.122:988: is it running a compatible version of Lustre? [12143.862739] LNetError: Skipped 1 previous similar message [12144.298528] Key type lgssc unregistered [12144.520838] LNet: 467616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12144.525507] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12144.529254] LNet: Removed LNI 192.168.206.122@tcp [12145.567280] LNet: Removed LNI 192.168.206.122@tcp100 [12145.569464] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12145.573611] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12145.581293] LNet: Removed LNI 192.168.206.122@tcp101 [12146.591262] LNet: Removed LNI 192.168.206.122@tcp102 [12146.593363] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12146.596757] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12146.607441] LNet: Removed LNI 192.168.206.122@tcp103 [12147.615311] LNet: Removed LNI 192.168.206.122@tcp104 [12148.639275] LNet: Removed LNI 192.168.206.122@tcp105 [12148.640819] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12148.649363] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12148.656245] LNet: Removed LNI 192.168.206.122@tcp106 [12149.663153] LNet: Removed LNI 192.168.206.122@tcp107 [12150.687185] LNet: Removed LNI 192.168.206.122@tcp108 [12151.711147] LNet: Removed LNI 192.168.206.122@tcp109 [12152.736038] LNet: Removed LNI 192.168.206.122@tcp110 [12152.741708] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12152.753382] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12152.760762] LNet: Removed LNI 192.168.206.122@tcp111 [12153.825258] LNet: Removed LNI 192.168.206.122@tcp112 [12154.847326] LNet: Removed LNI 192.168.206.122@tcp113 [12155.871091] LNet: Removed LNI 192.168.206.122@tcp114 [12156.896015] LNet: Removed LNI 192.168.206.122@tcp115 [12157.919154] LNet: Removed LNI 192.168.206.122@tcp116 [12158.943280] LNet: Removed LNI 192.168.206.122@tcp117 [12159.967163] LNet: Removed LNI 192.168.206.122@tcp118 [12160.991606] LNet: Removed LNI 192.168.206.122@tcp119 [12160.994685] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12161.003680] LNetError: 467616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12161.009733] LNet: Removed LNI 192.168.206.122@tcp120 [12161.012367] LNet: Removed LNI 192.168.206.122@tcp121 [12161.016804] LNet: Removed LNI 192.168.206.122@tcp122 [12161.020820] LNet: Removed LNI 192.168.206.122@tcp123 [12161.024053] LNet: Removed LNI 192.168.206.122@tcp124 [12161.026890] LNet: Removed LNI 192.168.206.122@tcp125 [12161.030347] LNet: Removed LNI 192.168.206.122@tcp126 [12161.033505] LNet: Removed LNI 192.168.206.122@tcp127 [12161.035861] LNet: Removed LNI 192.168.206.122@tcp128 [12161.038531] LNet: Removed LNI 192.168.206.122@tcp129 [12161.040577] LNet: Removed LNI 192.168.206.122@tcp130 [12161.043183] LNet: Removed LNI 192.168.206.122@tcp131 [12161.045080] LNet: Removed LNI 192.168.206.122@tcp132 [12161.047317] LNet: Removed LNI 192.168.206.122@tcp133 [12161.049778] LNet: Removed LNI 192.168.206.122@tcp134 [12161.061957] LNet: Removed LNI 192.168.206.122@tcp135 [12161.503823] Key type .llcrypt unregistered [12161.505987] Key type ._llcrypt unregistered [12174.748935] Key type ._llcrypt registered [12174.750631] Key type .llcrypt registered [12174.797616] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [12182.495552] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12182.850619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12182.890153] alg: No test for adler32 (adler32-zlib) [12183.768681] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12183.874252] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12185.471153] Key type lgssc registered [12185.912906] Lustre: Echo OBD driver; http://www.lustre.org/ [12201.943864] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:24:14 (1788852254) [12211.247396] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12213.231229] Key type lgssc unregistered [12213.520747] LNet: 471674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12213.524769] LNetError: 471674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12213.540071] LNet: Removed LNI 192.168.206.122@tcp [12214.129595] Key type .llcrypt unregistered [12214.132695] Key type ._llcrypt unregistered [12223.139244] Key type ._llcrypt registered [12223.141160] Key type .llcrypt registered [12223.197396] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12223.693108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12223.701168] alg: No test for adler32 (adler32-zlib) [12224.606416] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12224.741271] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12226.359151] Key type lgssc registered [12226.908518] Lustre: Echo OBD driver; http://www.lustre.org/ [12231.758548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12231.769908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12232.891247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12232.911891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12232.968077] Lustre: lustre-MDT0000: new disk, initializing [12233.011244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12233.020292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12234.981531] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12240.341903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.412386] Lustre: 472859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12240.448977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12240.454032] Lustre: Skipped 1 previous similar message [12240.496700] Lustre: lustre-MDT0001: new disk, initializing [12240.561892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12240.584734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12240.589139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12242.604181] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12247.694265] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12250.501983] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12253.611791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12253.751404] Lustre: lustre-OST0000: new disk, initializing [12253.754181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12253.758442] Lustre: 474826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12253.798061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.013464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12254.021154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12254.087174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12256.524649] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12261.772958] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12263.791596] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing lnet_if_list [12266.066632] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12266.080936] LNet: Added LNI 192.168.206.122@tcp10 [8/256/0/180] [12268.438157] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12268.454154] LNet: Added LNI 192.168.206.122@tcp11 [8/256/0/180] [12270.234652] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12270.255239] LNet: Added LNI 192.168.206.122@tcp12 [8/256/0/180] [12272.084924] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12272.103653] LNet: Added LNI 192.168.206.122@tcp13 [8/256/0/180] [12274.025788] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12274.039786] LNet: Added LNI 192.168.206.122@tcp14 [8/256/0/180] [12275.908569] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12275.925208] LNet: Added LNI 192.168.206.122@tcp15 [8/256/0/180] [12277.837619] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12277.850643] LNet: Added LNI 192.168.206.122@tcp16 [8/256/0/180] [12279.803370] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12279.819175] LNet: Added LNI 192.168.206.122@tcp17 [8/256/0/180] [12281.865191] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12281.877692] LNet: Added LNI 192.168.206.122@tcp18 [8/256/0/180] [12284.105496] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12284.121138] LNet: Added LNI 192.168.206.122@tcp19 [8/256/0/180] [12286.409611] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12286.427458] LNet: Added LNI 192.168.206.122@tcp20 [8/256/0/180] [12288.730326] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12288.748235] LNet: Added LNI 192.168.206.122@tcp21 [8/256/0/180] [12291.458511] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12291.476383] LNet: Added LNI 192.168.206.122@tcp22 [8/256/0/180] [12294.172298] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12294.188370] LNet: Added LNI 192.168.206.122@tcp23 [8/256/0/180] [12296.731296] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12296.747873] LNet: Added LNI 192.168.206.122@tcp24 [8/256/0/180] [12299.133671] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12299.150356] LNet: Added LNI 192.168.206.122@tcp25 [8/256/0/180] [12301.428347] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12301.445538] LNet: Added LNI 192.168.206.122@tcp26 [8/256/0/180] [12303.865673] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12303.880104] LNet: Added LNI 192.168.206.122@tcp27 [8/256/0/180] [12306.350787] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12306.366483] LNet: Added LNI 192.168.206.122@tcp28 [8/256/0/180] [12308.852799] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12308.868416] LNet: Added LNI 192.168.206.122@tcp29 [8/256/0/180] [12311.260792] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12311.275814] LNet: Added LNI 192.168.206.122@tcp30 [8/256/0/180] [12313.927904] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12313.945130] LNet: Added LNI 192.168.206.122@tcp31 [8/256/0/180] [12316.498541] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12316.513952] LNet: Added LNI 192.168.206.122@tcp32 [8/256/0/180] [12319.363339] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12319.379523] LNet: Added LNI 192.168.206.122@tcp33 [8/256/0/180] [12322.189251] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12322.203248] LNet: Added LNI 192.168.206.122@tcp34 [8/256/0/180] [12325.709804] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12325.733821] LNet: Added LNI 192.168.206.122@tcp35 [8/256/0/180] [12329.790583] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12329.807163] LNet: Added LNI 192.168.206.122@tcp36 [8/256/0/180] [12333.651700] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12333.668566] LNet: Added LNI 192.168.206.122@tcp37 [8/256/0/180] [12338.347322] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12338.381611] LNet: Added LNI 192.168.206.122@tcp38 [8/256/0/180] [12343.210932] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12343.232676] LNet: Added LNI 192.168.206.122@tcp39 [8/256/0/180] [12347.167731] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12347.196700] LNet: Added LNI 192.168.206.122@tcp40 [8/256/0/180] [12351.075119] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12351.095712] LNet: Added LNI 192.168.206.122@tcp41 [8/256/0/180] [12355.356802] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12355.384728] LNet: Added LNI 192.168.206.122@tcp42 [8/256/0/180] [12359.269206] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12359.292926] LNet: Added LNI 192.168.206.122@tcp43 [8/256/0/180] [12363.276650] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12363.290961] LNet: Added LNI 192.168.206.122@tcp44 [8/256/0/180] [12367.215464] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12367.237567] LNet: Added LNI 192.168.206.122@tcp45 [8/256/0/180] [12371.903643] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12371.934239] LNet: Added LNI 192.168.206.122@tcp46 [8/256/0/180] [12376.628724] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12376.644374] LNet: Added LNI 192.168.206.122@tcp47 [8/256/0/180] [12380.459772] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12380.477167] LNet: Added LNI 192.168.206.122@tcp48 [8/256/0/180] [12384.258564] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12384.275603] LNet: Added LNI 192.168.206.122@tcp49 [8/256/0/180] [12388.494285] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12388.509977] LNet: Added LNI 192.168.206.122@tcp50 [8/256/0/180] [12399.588093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12399.592194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.602365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12402.656125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12402.661860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12402.668879] Lustre: Skipped 1 previous similar message [12404.932190] Lustre: server umount lustre-MDT0000 complete [12406.640570] LustreError: 472851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852460 with bad export cookie 12854183366062087610 [12406.644372] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12406.646987] LustreError: 472851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12406.814779] Lustre: server umount lustre-MDT0001 complete [12414.727242] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12417.854662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12423.112141] Lustre: 472133:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852461/real 1788852461] req@000000006cf3bf92 x1875747566360192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12423.120635] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.167149] Lustre: 472134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852465/real 1788852465] req@000000001795951e x1875747566360320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12428.255177] Lustre: 472132:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852466/real 1788852466] req@00000000f7d57dea x1875747566360448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12431.327370] LustreError: 472130:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7b6d839 x1875747566361344/t0(0) o250->MGC192.168.206.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12431.431112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12432.479101] Lustre: 472131:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852470/real 1788852470] req@00000000777bf01f x1875747566360704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12432.814237] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12435.982173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.137492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12437.631276] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12438.176071] Lustre: 472133:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852476/real 1788852476] req@00000000729564fb x1875747566360832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12440.744529] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12441.575444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12443.625812] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12443.632829] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12453.860940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.871204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.874238] Lustre: Skipped 3 previous similar messages [12458.976088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12458.980542] Lustre: Skipped 1 previous similar message [12459.624088] Lustre: server umount lustre-OST0000 complete [12462.048155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12462.052930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.059195] Lustre: Skipped 1 previous similar message [12464.095974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.102741] Lustre: Skipped 2 previous similar messages [12467.618199] Lustre: server umount lustre-MDT0000 complete [12469.216258] LustreError: 499015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.225090] LustreError: 499015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12469.541568] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852523 with bad export cookie 12854183366062089871 [12469.544336] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12469.551088] LustreError: 472852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12469.686701] Lustre: server umount lustre-MDT0001 complete [12474.490880] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12475.762639] Key type lgssc unregistered [12475.956589] LNet: 501790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12475.959830] LNetError: 501790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12475.963634] LNet: Removed LNI 192.168.206.122@tcp [12475.965751] LNet: Removed LNI 192.168.206.122@tcp10 [12475.967852] LNet: Removed LNI 192.168.206.122@tcp11 [12475.969226] LNet: Removed LNI 192.168.206.122@tcp12 [12475.970703] LNet: Removed LNI 192.168.206.122@tcp13 [12475.972620] LNet: Removed LNI 192.168.206.122@tcp14 [12475.974224] LNet: Removed LNI 192.168.206.122@tcp15 [12475.976426] LNet: Removed LNI 192.168.206.122@tcp16 [12475.978407] LNet: Removed LNI 192.168.206.122@tcp17 [12475.981524] LNet: Removed LNI 192.168.206.122@tcp18 [12475.985639] LNet: Removed LNI 192.168.206.122@tcp19 [12475.987283] LNet: Removed LNI 192.168.206.122@tcp20 [12475.989848] LNet: Removed LNI 192.168.206.122@tcp21 [12475.991823] LNet: Removed LNI 192.168.206.122@tcp22 [12475.993977] LNet: Removed LNI 192.168.206.122@tcp23 [12475.995680] LNet: Removed LNI 192.168.206.122@tcp24 [12475.997070] LNet: Removed LNI 192.168.206.122@tcp25 [12475.999197] LNet: Removed LNI 192.168.206.122@tcp26 [12476.001367] LNet: Removed LNI 192.168.206.122@tcp27 [12476.003198] LNet: Removed LNI 192.168.206.122@tcp28 [12476.004698] LNet: Removed LNI 192.168.206.122@tcp29 [12476.006513] LNet: Removed LNI 192.168.206.122@tcp30 [12476.008161] LNet: Removed LNI 192.168.206.122@tcp31 [12476.009533] LNet: Removed LNI 192.168.206.122@tcp32 [12476.011239] LNet: Removed LNI 192.168.206.122@tcp33 [12476.012893] LNet: Removed LNI 192.168.206.122@tcp34 [12476.014052] LNet: Removed LNI 192.168.206.122@tcp35 [12476.015661] LNet: Removed LNI 192.168.206.122@tcp36 [12476.017203] LNet: Removed LNI 192.168.206.122@tcp37 [12476.021637] LNet: Removed LNI 192.168.206.122@tcp38 [12476.023878] LNet: Removed LNI 192.168.206.122@tcp39 [12476.025577] LNet: Removed LNI 192.168.206.122@tcp40 [12476.027197] LNet: Removed LNI 192.168.206.122@tcp41 [12476.029478] LNet: Removed LNI 192.168.206.122@tcp42 [12476.031237] LNet: Removed LNI 192.168.206.122@tcp43 [12476.032845] LNet: Removed LNI 192.168.206.122@tcp44 [12476.034169] LNet: Removed LNI 192.168.206.122@tcp45 [12476.036082] LNet: Removed LNI 192.168.206.122@tcp46 [12476.038117] LNet: Removed LNI 192.168.206.122@tcp47 [12476.040153] LNet: Removed LNI 192.168.206.122@tcp48 [12476.041953] LNet: Removed LNI 192.168.206.122@tcp49 [12476.054140] LNet: Removed LNI 192.168.206.122@tcp50 [12476.479459] Key type .llcrypt unregistered [12476.481430] Key type ._llcrypt unregistered [12486.394081] Key type ._llcrypt registered [12486.395724] Key type .llcrypt registered [12486.441319] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [12494.093246] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12494.417688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12494.436920] alg: No test for adler32 (adler32-zlib) [12495.356964] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12495.486627] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12497.111168] Key type lgssc registered [12497.659507] Lustre: Echo OBD driver; http://www.lustre.org/ [12512.085099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12512.093981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12513.180861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12513.197781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12513.239440] Lustre: lustre-MDT0000: new disk, initializing [12513.266412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12513.274413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12514.814150] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12519.586996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12519.641755] Lustre: 504956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12519.667376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12519.669785] Lustre: Skipped 1 previous similar message [12519.708089] Lustre: lustre-MDT0001: new disk, initializing [12519.753696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12519.773435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12519.779923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12521.365280] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12525.556847] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12527.889940] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12531.351465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12531.435319] Lustre: *** cfs_fail_loc=90f, val=0*** [12531.467109] Lustre: lustre-OST0000: new disk, initializing [12531.469767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12531.473373] Lustre: 506975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12531.512625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12534.094137] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12539.171262] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12539.420581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12539.428222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12539.466145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12544.483554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.485865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12544.485972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12544.489601] Lustre: Skipped 1 previous similar message [12544.498599] Lustre: Skipped 1 previous similar message [12548.510774] Lustre: server umount lustre-OST0000 complete [12550.623806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12550.628022] LustreError: Skipped 1 previous similar message [12550.629756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12550.636745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12554.720580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12554.721498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.281683] Lustre: server umount lustre-MDT0000 complete [12558.057379] LustreError: 505754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852612 with bad export cookie 8377286354538816711 [12558.059565] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12558.061717] LustreError: 505754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12558.234313] Lustre: server umount lustre-MDT0001 complete [12563.654848] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12565.009909] Key type lgssc unregistered [12565.161525] LNet: 508876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12565.164813] LNetError: 508876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12565.175562] LNet: Removed LNI 192.168.206.122@tcp [12565.592539] Key type .llcrypt unregistered [12565.594344] Key type ._llcrypt unregistered [12575.402754] Key type ._llcrypt registered [12575.404645] Key type .llcrypt registered [12575.465996] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12575.862157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12575.871176] alg: No test for adler32 (adler32-zlib) [12576.758198] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12576.864414] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12578.471172] Key type lgssc registered [12578.933951] Lustre: Echo OBD driver; http://www.lustre.org/ [12581.843481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12581.849371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.983344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12584.373597] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12587.492200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12587.631336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12589.103678] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12591.866915] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12593.911041] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12596.458090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12596.591584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12598.841310] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12601.847712] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12616.813516] loop0: detected capacity change from 0 to 204800000 [12616.896901] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12616.903661] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12616.910185] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12619.387109] loop0: detected capacity change from 0 to 204800000 [12619.460875] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12619.467646] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12619.474062] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12622.036277] loop0: detected capacity change from 0 to 204800000 [12624.984914] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12625.083392] Lustre: fs15246-OST0000: new disk, initializing [12625.087332] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12625.091080] Lustre: 513730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12625.118087] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12627.766414] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12632.644562] loop1: detected capacity change from 0 to 204800000 [12635.786501] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.869455] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12635.916540] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12635.971819] Lustre: fs15246-MDT0000: new disk, initializing [12636.077145] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12636.097303] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12636.158109] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12637.217550] Lustre: 514850:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009fc6f2c0 x1875747935389312/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788852707 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12638.600231] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12646.307626] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12646.315629] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12646.334308] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12653.030826] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.035941] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12655.397216] Lustre: server umount fs15246-MDT0000 complete [12658.148704] LustreError: 512607:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.156334] LustreError: 512607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12659.679130] Lustre: server umount fs15246-OST0000 complete [12668.386620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.388555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12668.392925] Lustre: Skipped 1 previous similar message [12668.398828] Lustre: Skipped 1 previous similar message [12670.433789] Lustre: server umount lustre-OST0000 complete [12673.505761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12673.512056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12673.513733] Lustre: Skipped 1 previous similar message [12673.518896] Lustre: Skipped 1 previous similar message [12678.298624] Lustre: server umount lustre-MDT0000 complete [12678.624336] LustreError: 510112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.631773] LustreError: 510112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12679.841315] LustreError: 510098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852733 with bad export cookie 9325591594163263999 [12679.844281] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12679.847314] LustreError: 510098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12679.978812] Lustre: server umount lustre-MDT0001 complete [12685.847490] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12687.593742] Key type lgssc unregistered [12687.751719] LNet: 517940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12687.757162] LNetError: 517940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12687.768709] LNet: Removed LNI 192.168.206.122@tcp [12688.294674] Key type .llcrypt unregistered [12688.296301] Key type ._llcrypt unregistered [12699.836112] Key type ._llcrypt registered [12699.837781] Key type .llcrypt registered [12699.893317] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12700.359439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12700.399344] alg: No test for adler32 (adler32-zlib) [12701.319247] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [12701.464909] LNet: Added LNI 192.168.206.122@tcp [8/256/0/180] [12703.079169] Key type lgssc registered [12703.547230] Lustre: Echo OBD driver; http://www.lustre.org/ [12717.365460] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [12721.438753] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12736.828427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12736.837731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.951433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12737.967238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12738.011801] Lustre: lustre-MDT0000: new disk, initializing [12738.055034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12738.069309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12739.943062] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12745.757726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12745.824193] Lustre: 522371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12745.861767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12745.864711] Lustre: Skipped 1 previous similar message [12745.917572] Lustre: lustre-MDT0001: new disk, initializing [12745.970485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12745.995179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12746.005769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12748.160424] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12754.005970] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12757.780320] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12761.612659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12761.774265] Lustre: lustre-OST0000: new disk, initializing [12761.777136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12761.781915] Lustre: 524340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12761.836354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12764.880750] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug -1 all [12769.561566] Lustre: DEBUG MARKER: oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12770.827262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12770.831895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12770.889507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12771.411188] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12771.504130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12773.256864] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12773.363238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12775.905785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12775.908252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.908598] LustreError: Skipped 1 previous similar message [12775.914853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12775.918353] Lustre: Skipped 1 previous similar message [12780.384830] Lustre: server umount lustre-OST0000 complete [12782.047957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12782.050897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.055303] Lustre: Skipped 1 previous similar message [12782.057312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.144573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.147345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.265339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12791.268675] Lustre: Skipped 2 previous similar messages [12795.360481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12795.362701] Lustre: Skipped 1 previous similar message [12796.383234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12796.464694] Lustre: server umount lustre-MDT0000 complete [12798.351686] LustreError: 522365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852852 with bad export cookie 12456963247834934377 [12798.357130] LustreError: MGC192.168.206.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12798.358657] LustreError: 522365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12798.529114] Lustre: server umount lustre-MDT0001 complete [12810.254750] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_hostid [12814.350786] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing load_modules_local [12828.452563] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12548 sec ========= 03:34:41 (1788852881) [12829.283626] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:34:42 (1788852882) === [12830.945555] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:34:43 (1788852883) === [12846.694848] Lustre: DEBUG MARKER: oleg622-server.virtnet: executing unload_modules_local [12847.774036] Key type lgssc unregistered [12847.905548] LNet: 530973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12847.909468] LNetError: 530973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12847.916637] LNet: Removed LNI 192.168.206.122@tcp [12848.257956] Key type .llcrypt unregistered [12848.259266] Key type ._llcrypt unregistered